How to monitor container-name/ID on docker swarm architecture on multiple nodes ?

zouzou33 commented 1 year ago

⚠️ Please verify that this bug has NOT been raised before.

[X] I checked and didn't find similar issue

🛡️ Security Policy

[X] I agree to have read this project Security Policy

📝 Describe your problem

Hello,

Sorry in advance if I make any mistake or if my question is obvious to you. I am new on github, uptime kuma and pretty new on Docker Swarm as well. I am spent a lot of time to find out a solution but I'm blocked so here I am.

First of all here is my architecture : Docker-swarm with 1 manager et 2 workers. Each node (even manager) run some containers. I use two overlay networks(1 frontend ,1 backend) and the ingress mesh routing by defaults. I also use glusterFs to share volume across different nodes.

+------------+   +-------------+   +-------------+
|   Manager  |   |   Worker 1  |   |   Worker 2  |
|+----------+|   |+-----------+|   |+-----------+|
||uptime-kuma||  || uptime-kuma||  || uptime-kuma||
|| Container1||  || Container3 ||  || Container5 ||
|| Container2||  || Container4 ||  || Container6 ||
|+---------+|    |+-----------+|   |+-----------+|
+-----------+    +-------------+   +-------------+
     \/                 \/               \/ 
 +------------------------------------------------+
 |                  Volumes                       |
 |                 (GlusterFS)                    |
 |                                                |
 |              /mnt/docker-data/ on another hdd  |
 +------------------------------------------------+

What I want to do ? I would like to be able to monitor all my containers through their Name/ID whatever which node they are running. I am talking about the menu "Monitor Type" -> "Docker Container" -> "Container Name/ID". I have a lack of knowledge. I don't know if what I want to do is feasible. Currently what I understand is : if uptime-kuma is deployed on the manager it will be able to monitor via the "container_name" only containers hosted on the manager because the "/var/run/docker.sock" is shared only with manager host.

I have tried to use the mode: global in my stack but same behavior, i am only able to monitor container_name on the manager but not on Worker 1 & Worker2.

Here below, an exemple of my docker stack

herebelow my docker stack

version: '3.8'

services: uptime-kuma: image: louislam/uptime-kuma:1 container_name: uptime-kuma volumes:

/mnt/docker-data/uptime-kuma-data:/app/data
/var/run/docker.sock:/var/run/docker.sock ports:
3001:3001
restart: always deploy: mode: global networks:
test_frontend
test_backend

networks: test_frontend: external: true test_backend: external: true

Thanks in advance for your lights.

📝 Error Message(s) or Log

No response

🐻 Uptime-Kuma Version

1.21.2

💻 Operating System and Arch

Ubuntu 22.04.2 LTS 64bit

🌐 Browser

Firefox 111.0..1

🐋 Docker Version

23.0.3

🟩 NodeJS Version

No response

chakflying commented 1 year ago

From their documentation it looks like you should be able to access containers running on other nodes in the same swarm if you query the manager. Have you tried to use the full container ID in the monitor? Or maybe I'm reading their doc incorrectly.

zouzou33 commented 1 year ago

Hello, I have tried to do an api call from the manager (cf attachement) but containers outside the manager is not detected. Best regards, api_docker

zouzou33 commented 1 year ago

Hello, Allow me to update this post. So, no one has an idea to do what I explain in my initial post?

Is it even possible for a swarm architecture? Thanks you.

qnlbnsl commented 1 year ago

Hi, is there any update on this?

imoize commented 1 year ago

I'm running docker swarm with 3 nodes and Ceph as storage distribution but I don't have any problem with database running on CephFS.

I had the same problem a few weeks ago, but only the monitoring part in the docker swarm cluster to monitor the service, in docker swarm I can't use the container name because sometimes the name changes.

To solve this problem I changed the Monitor Type to PING or TCP

And the Hostname to the Service name.

To find out the service name by running:

docker service ls

To use this method I have to use Adguard or Pihole DNS server, otherwise the ISP will try to resolve the Hostname and reply with a random ip. so the Monitor will detect that the container is still running, even though it is not.

So far it is still working fine in Swarm mode until now. And reports if the container is Down or Up correctly.

This may not be the right way to do this. Thanks

djn84 commented 1 year ago

I could never get the docker integration working due to container ID/names changing constantly with a swarm deployment, but discovered a decent workaround:

Using HTTP - Keyword monitors that query the swarm task states of the docker services...

URL http://<manager node ip>:2375/tasks?filters={"desired-state":["running"],"service":["service_name"]}

Keyword "State":"running"

That way if the running task fails on that service then the monitor will no longer be able to find one in a running state and the .

scyto commented 6 months ago

I could never get the docker integration working due to container ID/names changing constantly with a swarm deployment, but discovered a decent workaround

This works well for a service with only 1 replica

However when there are 3 replicas its breaks down as, say, one replica failing will mean the HTTP response has 2 instances of state:running and one of state:failed in the output.

Do you have a suggestion on how to mitigate?

CommanderStorm commented 6 months ago

If the response of said api is json, you could use the json-query monitor. This might also be a good solution (have not checked it out) https://github.com/louislam/uptime-kuma/issues/957#issuecomment-1879761339

louislam / uptime-kuma