search

stefanprodan / swarmprom

Docker Swarm instrumentation with Prometheus, Grafana, cAdvisor, Node Exporter and Alert Manager
MIT License
1.87k stars 735 forks source link

How to Monitor Nodes outside the swarm cluster. #57

Open Neru007 opened 6 years ago

Neru007 commented 6 years ago

I'm trying to monitor few nodes outside the swarm cluster but unable to reach those nodes from inside the prometheus container

hatched-DavidMichon commented 6 years ago

Hi @stefanprodan, thanks for your stack. I come up with a simialr problem as @Neru007 and didn't find out the best approach to take.

Actually I have 3 swarm clusters, all separated in different VPC on AWS. One of the swarm is for the tools (Prometheus, Grafana, Jenkins, and so on). The 2 others are production and staging.

I would like to scrape metrics of my production and staging stack from prometheus but I didn't find out how to have this setup dynamically. When on the same swarm cluster, I use the dockerflow/docker-flow-swarm-listener that communicate with dockerflow/docker-flow-monitor to automatically reconfigure the scrape config. But this work only within the same swarm cluster as it takes the internal swarm IP of the container to configure the scrape.

Do you think your stack can overcome this limitation? Or do you have any idea how this can be resolved without manually set a scrape config?

yuklia commented 6 years ago

@hatched-DavidMichon did you solve this issue?

hatched-DavidMichon commented 6 years ago

@yuklia Not really as I did a workaround by adding manual scrape configs for my production and staging stack.

On AWS I first created private hosted zones for my production and staging stack. On those hosted zones I created SRV entries for my exporters (node-exporters, cadvisor, etc..). Then I created scrape config that point to those SRV entries to dynamically get the list of internal node IPs so prometheus can scrape metrics from them.

This is not ideal because, even if the list of IPs (on the SRV entries) are build up dynamically as I add/remove nodes from my clusters, if dockerflow/docker-flow-monitor restart, it does not get alerts setup from those 2 stacks because it looks only on the current swarm cluster it's running on.

To have my alerts from production and staging I have to restart dockerflow/docker-flow-swarm-listener on those 2 stacks so it send the alerts config to prometheus.

Ideally dockerflow/docker-flow-monitor should accept a list of LISTENER_ADDRESS to call dockerflow/docker-flow-swarm-listener on multiple hosts. Same for the DF_GET_NODES_URL.