search

stefanprodan / swarmprom

Docker Swarm instrumentation with Prometheus, Grafana, cAdvisor, Node Exporter and Alert Manager
MIT License
1.87k stars 735 forks source link

Monitoring Swarm with multiple masters #115

Open monsoft opened 5 years ago

monsoft commented 5 years ago

I have simple 3 nodes swarm cluster where each node is master and worker:

        "oxmqs5lze5dk5m6ni0me4msnu *   dock1               Ready               Active              Leader              19.03.1",
        "cwyprb7cxtepai541h58no4as     dock2               Ready               Active              Reachable           19.03.1",
        "o3b2ttd0wxbyfh1e1yovj8rxw     dock3               Ready               Active              Reachable           19.03.1"

So because in current docker-compose.yml file placement for Grafana, Prometheus, Caddy and AlertManager is setup to be deployed on master, I changed it to be deployed only to dock1 using placement

placement:
  constraints:
    - node.hostname == dock1

Unfortunately it didn't work and most of these services went to dock3. Next try I applied label "Monitoring=true" to dock1 and used placement :

placement:
  constraints:
    - node.labels.Monitoring == true

but this same situation :(

When I did this same on Swarm node with 1 master only, all placement constraints works correctly. Does anyone have this same situation like I have ?