search

stefanprodan / dockprom

Docker hosts and containers monitoring with Prometheus, Grafana, cAdvisor, NodeExporter and AlertManager
MIT License
6.02k stars 1.72k forks source link

HTTP Error Bad Gateway #213

Closed erichegit closed 3 years ago

erichegit commented 3 years ago

I followed 1:1 the Readme - all Containers are up and running. But I receive "HTTP Error Bad Gateway" in Grafana when I look at my default Datasource.

Browsing to IP:9090 shows me 502 Bad Gateway

How can I fix this?

Grafana http://:3000 = Working Prometheus http://:9090 = BAD Gateway Prometheus-Pushgateway http://:9091 = Working AlertManager http://:9093 = BAD Gateway

levesduzw commented 3 years ago

Had this issue as well, I believe it has something to do with the default Docker IP subnet.

Fix: when I did a docker inspect prometheus | grep IPAddress on the Prometheus container to get its IP, and set that in Grafana's Data source config for Prometheus, it worked. e.g. docker logs grafana showed that it was trying to query "127.0.0.1:9090". The IP that worked was "172.24.0.1:9090". Note that this can change when you redeploy.

I'm not sure what circumstances are required on a server for this to work out of the box.

nightah commented 3 years ago

The data source that Grafana provisions for Prometheus shouldn't reference 127.0.0.1.

https://github.com/stefanprodan/dockprom/blob/5cc61d4c4c58386cb38e89bf18f80e19746ec379/grafana/provisioning/datasources/datasource.yml#L8

It should be referencing Prometheus by its DNS name.

levesduzw commented 3 years ago

Then this is a DNS configuration issue, at least for me. nslookup is supposed to work in the container, right? It's not working here, with nameserver 127.0.0.11. It could be that whatever method Grafana uses to resolve prometheus also fails. However ping, wget, arp do resolve prometheus from the Grafana container.

My host is a company server so I can't modify /etc/resolv.conf to troubleshoot. It does include ("search") some company domains by default, which might cause local lookups to fail.

According to https://docs.docker.com/config/containers/container-networking/:

containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host

Is there an easy way to avoid putting the host's search domains in the containers?

nightah commented 3 years ago

nslookup fails for me too, however I do not experience the same issues. Grafana seems to still be able to resolve prometheus to it's respective Docker network IP:

/usr/share/grafana $ ping prometheus
PING prometheus (172.19.0.8): 56 data bytes
ping: permission denied (are you root?)
/usr/share/grafana $ nslookup prometheus
Server:     127.0.0.11
Address:    127.0.0.11:53

** server can't find prometheus.tokyo-3.vpn: NXDOMAIN

** server can't find prometheus.tokyo-3: NXDOMAIN

** server can't find prometheus.guest: NXDOMAIN

** server can't find prometheus.tokyo-3.wlan: NXDOMAIN

** server can't find prometheus.tokyo-3.vpn: NXDOMAIN

** server can't find prometheus.cygnus: NXDOMAIN

** server can't find prometheus.tokyo-3: NXDOMAIN

** server can't find prometheus.tokyo-3.wlan: NXDOMAIN

** server can't find prometheus.guest: NXDOMAIN

** server can't find prometheus.cygnus: NXDOMAIN

** server can't find prometheus.goomba: NXDOMAIN

** server can't find prometheus.goomba: NXDOMAIN

** server can't find prometheus.eva01: NXDOMAIN

** server can't find prometheus.eva01: NXDOMAIN

So I'm not sure if this is a red herring but I don't think that's the actual issue that you may be experiencing.

nightah commented 3 years ago

I've also tried running this up in a brand new VM with no modifications without any issues.

Have you tried a fresh clone?

nightah commented 3 years ago

Closing as I was not able to reproduce this. If you continue to have issues please request for this to be reopened and provide steps to reproduce the problem.