search

f5devcentral / application-study-tool

The Application Study Tool provides enhanced insights into (classic) BIG-IP products, leveraging best in class open source telemetry tools
Apache License 2.0
17 stars 6 forks source link

Prometheus not starting in Docker container, able to get to Grafana tho #2

Closed ezzd121 closed 1 month ago

ezzd121 commented 1 month ago

When attempting to start the application study tool on my linux server, Grafana Starts, but Prometheus is failing. I am able to get to the port for Grafana. Port 9090 comes up then drops almost immediately

application-study-tool-otel-collector-1 | 2024-09-18T16:46:59.825Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/metrics", "error": "failed to make an HTTP request: Post \"http://prometheus:9090/api/v1/otlp/v1/metrics\": read tcp 172.18.0.4:38378->10.16.6.24:9090: read: connection reset by peer", "interval": "5.037975262s"} application-study-tool-otel-collector-1 | 2024-09-18T16:47:04.422Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/metrics", "error": "failed to make an HTTP request: Post \"http://prometheus:9090/api/v1/otlp/v1/metrics\": dial tcp 172.18.0.3:9090: connect: connection refused", "interval": "33.864628334s"} prometheus | ts=2024-09-18T16:47:04.465Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" prometheus | ts=2024-09-18T16:47:04.465Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" prometheus exited with code 2 application-study-tool-otel-collector-1 | 2024-09-18T16:47:04.992Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/metrics", "error": "failed to make an HTTP request: Post \"http://prometheus:9090/api/v1/otlp/v1/metrics\": read tcp 172.18.0.4:38384->10.16.6.24:9090: read: connection reset by peer", "interval": "5.80800503s"} application-study-tool-otel-collector-1 | 2024-09-18T16:47:06.145Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/metrics", "error": "failed to make an HTTP request: Post \"http://prometheus:9090/api/v1/otlp/v1/metrics\": read tcp 172.18.0.4:38396->10.16.6.24:9090: read: connection reset by peer", "interval": "33.454911972s"}

Docker Version docker -v Docker version 20.10.11, build dea9396 [ ~]$ netstat -plunt (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:5201 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:3000 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:4317 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:35391 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:9090 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:10050 0.0.0.0: LISTEN -
udp 0 0 0.0.0.0:111 0.0.0.0: -
udp 0 0 127.0.0.1:323 0.0.0.0: -
[ ~]$ [ ~]$ netstat -plunt (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:5201 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:3000 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:4317 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:35391 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:10050 0.0.0.0: LISTEN -
udp 0 0 0.0.0.0:111 0.0.0.0: -
udp 0 0 127.0.0.1:323 0.0.0.0:* -
[ ~]$

clhain commented 1 month ago

Thanks for the report - can you attach the logs for the prometheus container if it spits anything out before it crashes? Something like docker logs application-study-tool_prometheus_1 (container name may be different).

ezzd121 commented 1 month ago

this is the output I am getting:

ts=2024-09-19T15:24:49.172Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:24:49.172Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:25:49.669Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:25:49.669Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:26:50.179Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:26:50.179Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:27:50.716Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:27:50.716Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:28:51.229Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:28:51.229Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:29:51.721Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:29:51.721Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:30:52.254Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:30:52.254Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:31:52.765Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:31:52.765Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:32:53.286Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:32:53.286Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:33:53.822Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:33:53.822Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:34:54.361Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:34:54.362Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:35:54.832Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:35:54.832Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:36:55.347Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:36:55.347Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:37:55.849Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:37:55.849Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:38:56.376Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:38:56.376Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:39:56.897Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:39:56.897Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:40:57.423Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:40:57.423Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:41:57.929Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:41:57.929Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:42:58.473Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:42:58.473Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:30.490Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:30.490Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:31.132Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:31.132Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:31.916Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:31.916Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:32.837Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:32.837Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:34.149Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:34.149Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:36.246Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:36.246Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied" ts=2024-09-19T15:46:39.967Z caller=main.go:184 level=info msg="Experimental OTLP write receiver enabled" ts=2024-09-19T15:46:39.967Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: permission denied"

clhain commented 1 month ago

Awesome thanks! So it looks like it's not liking the file permission on the config file in ./services/prometheus/prometheus.yml... can you double check what they're set to? Should just need read access

ezzd121 commented 1 month ago

well, yup.. just changed the perms to 444 and it started up..

clhain commented 1 month ago

Great! Haven't seen this before, but I would also double check that all of the files under the service directory are readable... you should see a bunch of dashboards in the Grafana instance and if not it might be the permissions there too. Will go ahead and close this out.