search

vfarcic / docker-flow-stacks

85 stars 60 forks source link

Wrong prometheus endpoint? #9

Closed katopz closed 7 years ago

katopz commented 7 years ago

After deploy prometheus-grafana-df-proxy.yml

I found that at page http://localhost:9091/targets
has prometheus endpoint
http://localhost:9090/metrics
but it link to
http://b1b1472b3e3e:9090/metrics which look like node id instead of node ip.

image

And actually it should be http://localhost:9091/metrics? (It did print output fine at 9091)

Thanks

vfarcic commented 7 years ago

The address should be localhost:9090 since that's the port Prometheus is listening to. 9091 is the mapping for the requests coming from outside the container. Since the target looks for localhost, it should be the internal port 9090.

I did notice the same error before but it disappeared a bit later when it tried to scrape it again. Can you double check that's not your case?

I rerun that stack a few minutes ago and go everything green. If in your case prometheus endpoint continues being red after a while (give it a few minutes just in case), we should try to figure out what's the difference between my system and yours.

katopz commented 7 years ago

For some reason it's appear all green now after remove all node and redo all over again, I can't tell what go wrong there because I try too many thing at that time even reinstall docker and also restart.

One thing I notice is Docker somehow provide different IP after many remove all nodes and retry, usually it always like...

node-1 = 192.168.99.100 node-2 = 192.168.99.101 node-3 = 192.168.99.102

but somehow it was like...

node-1 = 192.168.99.106 node-2 = 192.168.99.107 node-3 = 192.168.99.108

And start to have this issue (and solved by that comment) in between.

I'll close this in case for now in case I can't reproduce it any more.

Thanks