Prometheus config file a bit misleading

[suntereo]

The grayling.conf file for this repo seems to suggest that Prometheus is automatically included. However, Prometheus is not automatically installed in the Docker version. You can see this by comparing the Docker version with the non-Docker version.

This led to hours of confusion on my end. Poor me, I know. I would like to document what I did to get Prometheus working inside the Docker build. And maybe somebody can remove the Prometheus settings from the grayling.conf file. Or better, bring Prometheus into this build as well.

Here's how I installed the Prometheus Exporter for the Docker version

1. Created a folder called GraylogPluginFolder on my local Mac.
2. Copied metrics-reporter-prometheus-3.0.0.jar from here into this folder.
3. Modified Docker Compose file to read from the plugins folder:

   graylog:
   image: graylog/graylog:4.2
   volumes:
     - /Users/shawn/GrayLog/GraylogPluginFolder:/usr/share/graylog/plugin # <-- **THIS**
   environment:
     - GRAYLOG_PASSWORD_SECRET=hfghsfgjfgjfhjghkj
     # Password: admin
     - GRAYLOG_ROOT_PASSWORD_SHA2=2378654782568726513459314
     - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
     # don't try below; does not apply to Docker version because Prometheus is not automatically included
     # - prometheus_exporter_enabled=true
   links:
     - mongodb:mongo
     - elasticsearch
   depends_on:
     - mongodb
     - elasticsearch
   ports:
     - 9000:9000 # Graylog web interface and REST API
     - 9090:9090
     - 5555:5555 # Raw/Plaintext TCP
     - 1514:1514 # Syslog TCP
     - 1514:1514/udp # Syslog UDP
     - 12201:12201 # GELF TCP
     - 12201:12201/udp # GELF UDP

4. Restarted Docker image and could now see in the startup logs:

   2022-08-24 22:03:02,476 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Internal Metrics Prometheus Reporter 3.0.0 [org.graylog.plugins.metrics.prometheus.MetricsPrometheusReporterMetaData]

5. Went to this URL once Graylog was running: http://localhost:9000/api/plugins/org.graylog.plugins.metrics.prometheus/metrics

Hopefully that will help some struggling soul who is pulling their hair out over this!

[supahgreg]

As of 4.1 Graylog has a built-in Prometheus exporter that should be used instead of the archived project you mentioned. This isn't anything specific to the Docker image.

Try adding these environment variables:

• GRAYLOG_PROMETHEUS_EXPORTER_ENABLED=true
• GRAYLOG_PROMETHEUS_EXPORTER_BIND_ADDRESS=0.0.0.0:9833

... and then add port 9833:9833/tcp and ~restart~ rebuild. After that you should be able to hit http://$SOME_HOST_IP:9833/ and see metrics.

[suntereo]

Thank you Greg! I can confirm your steps do work on my end.

But I'm still confused. The graylog.conf file shows 127.0.0.1:

# IP address and port for the Prometheus exporter HTTP server.
# Default: 127.0.0.1:9833
#prometheus_exporter_bind_address = 127.0.0.1:9833

Because of this, 127.0.0.1 is what I was trying to use. But this fails on my end with:

Your advice to use 0.0.0.0 does resolve this error.

But if you use 127.0.0.1:9833, does it work on your end? If not, shouldn't graylog.conf be updated to show 0.0.0.0?

[supahgreg]

The difference is that you're running Graylog within a Docker container rather than directly on the host system. By default, 127.0.0.1 in your container is local to the container (i.e. it's not the 127.0.0.1 of your host system). 0.0.0.0 binds to all available interfaces in the container, including the one used when exposing ports. Docker's networking documentation covers all of this in detail.

127.0.0.1:9833 is a reasonable default, as it prevents unintentionally exposing metrics.

[suntereo]

Thanks again. That is very helpful.

I bet others will blindly follow the comments in the graylog.conf and have it fail like I did. But I suppose they can just find this thread and get the solution they need.