Swarm

[xeor]

First; VERY nice put toghter! Thank you very much for this!

I must admit, I don't have much experience with prometheus, but how would a swarm-mode setup work with this project? Would it be as easy as setting up collectors on all node and have the monitor network on overlay?

ps: Sorry for submitting this as an issue.. It is sort of a feature request 👍

[stefanprodan]

Hello @xeor

When running the Prometheus stack on Docker Swarm you need to decide if you want to use the same Prometheus database for all Swarm nodes. If that's the case, then you need to do the following:

• change the monitor network to overlay
• run Prometheus and Grafana services as a single replica and pin them to a single node (since Prometheus and Grafana need to persist data on the host you can't allow Swarm to redistribute them to a random node)
• run Alertmanager service as a single replica (since this service is stateless, it doesn't matter if Swarm moves it around, but you need to embed the Alertmanager config in the container)
• run NodeExporter and cAdvisor services as global services (Swarm will set one replica of each service on each node)

The challenge with this setup is that Swarm will load balance the Prometheus scraping calls to NodeExporter and cAdvisor. So you can't use the host name of these 2 services in the Prometheus config. Docker Swarm doesn't have a Prometheus discovery integration like Kubernetes has. I know that Docker plans to do this, but right now it's not available.

A workaround would be to use the Prometheus dns_sd_config for service discovery, or you could deploy a dedicated NodeExporter and cAdvisor service for each Swarm node and use the node constrain to prevent Swarm from moving your services between nodes.

I plan to extend dockprom to Docker Swarm in the future.

[xeor]

Thanks for a very good answer, it's all logical now :)

Looking forward to a future update..! Keep it up!

[larryboymi]

I had to change a few of the queries on the container grafana dashboard to do sums, but now it's working pretty decently. I'm still trying to figure out how to get container (or service) names to show up instead of the generated id's, but wonder if that might be a cAdvisor/newest swarm issue. Thanks for your tips.

[alexsandro-xpt]

Swarm Mode +1

[Hermain]

"A workaround would be to use the Prometheus dns_sd_config for service discovery" Could you please explain what speaks against this workaround? It seems to solve the problem of the swarm loadbalancing the requests.

[stefanprodan]

@Hermain if you use the dns for node-exporter than you'll not be able to link the source to the hosts. every time you'll restart node-export it will get another IP, how will you match the source to the real host name?

[Hermain]

@stefanprodan I only wanted to understand why you call it a workaround and not a solution. Now I understand. The problem is, you can't link the node-exporter instance to a host. Thank you for the explanation.

[stefanprodan]

One way I could think of is to inject a label with the Docker Swarm Node ID on each metric in node-exporter. This way, even if the node-exporter changes IP you have the same ID, since this is generated at Swarm init. I'm trying to port dockprom to swarm this week and I'll post on the blog if I manage to run it properly.

[vguna]

Sounds promising!

[pascalandy]

IMHO, we could split Swarm Version to another project. At this point, I assume this project is for a single host monitoring

[imajes]

pinging in to see where this might go... also interested.