search

monitoringartist / zabbix-docker-monitoring

:whale: Docker/Kubernetes/Mesos/Marathon/Chronos/LXC/LXD/Swarm container monitoring - Docker image, Zabbix template and C module
https://hub.docker.com/r/monitoringartist/zabbix-agent-xxl-limited/
GNU General Public License v2.0
1.19k stars 266 forks source link

Cannot detect docker stat directory / no stat directory #175

Open labanana34 opened 2 years ago

labanana34 commented 2 years ago

Hello Jan Garaj,

If I understand correctly you are the one who's running this place :) May I have your help ?

I'm trying to make your module work on Ubuntu 22.04, so I compile the module following your docs:

apt-get install -y wget autoconf automake gcc git make pkg-config libpcre3-dev libjansson-dev
git clone -b 4.0.40 --depth 1 https://github.com/zabbix/zabbix.git /usr/src/zabbix
or
git clone -b 5.0.17 --depth 1 https://github.com/zabbix/zabbix.git /usr/src/zabbix
cd /usr/src/zabbix
./bootstrap.sh
./configure --enable-agent
mkdir src/modules/zabbix_module_docker
cd src/modules/zabbix_module_docker
wget https://raw.githubusercontent.com/monitoringartist/zabbix-docker-monitoring/master/src/modules/zabbix_module_docker/zabbix_module_docker.c
wget https://raw.githubusercontent.com/monitoringartist/zabbix-docker-monitoring/master/src/modules/zabbix_module_docker/Makefile
make

I made my test with zabbix agent 4.0.40 and 5.0.17 But I end up with the same error.

Here the full log of zabbix agent startup:

 40752:20220525:163347.520 Zabbix Agent stopped. Zabbix 5.0.17 (revision 0a4ac3dabc).
 40832:20220525:163347.559 Starting Zabbix Agent [box-oraXX]. Zabbix 5.0.17 (revision 0a4ac3dabc).
 40832:20220525:163347.559 **** Enabled features ****
 40832:20220525:163347.559 IPv6 support:          YES
 40832:20220525:163347.559 TLS support:           YES
 40832:20220525:163347.559 **************************
 40832:20220525:163347.559 using configuration file: /etc/zabbix/zabbix_agentd.conf
 40832:20220525:163347.559 In zbx_load_modules()
 40832:20220525:163347.559 loading module "/usr/lib/zabbix/modules/zabbix_module_docker.so"
 40832:20220525:163347.560 In zbx_module_api_version()
 40832:20220525:163347.560 In zbx_module_init()
 40832:20220525:163347.560 zabbix_module_docker v0.7.0, compilation time: Feb 22 2022 16:52:10
 40832:20220525:163347.560 In zbx_docker_dir_detect()
 40832:20220525:163347.560 Cannot detect docker stat directory
 40832:20220525:163347.560 In zbx_docker_api_detect()
 40832:20220525:163347.560 In zbx_docker_perm()
 40832:20220525:163347.561 zabbix agent user has docker perm
 40832:20220525:163347.561 In zbx_module_docker_socket_query()
 40832:20220525:163347.561 Docker's socket query: GET /_ping HTTP/1.0
 40832:20220525:163347.562 Docker's socket response: OK
 40832:20220525:163347.562 Docker's socket works - extended docker metrics are available
 40832:20220525:163347.562 In zbx_module_item_list()
 40832:20220525:163347.562 In zbx_module_item_timeout()
 40832:20220525:163347.562 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_docker.so": /usr/lib/zabbix/modules/zabbix_module_docker.so: undefined symbol: zbx_module_history_write_cbs
 40832:20220525:163347.562 loaded modules: zabbix_module_docker.so
 40832:20220525:163347.562 End of zbx_load_modules():SUCCEED
 40832:20220525:163347.562 In init_collector_data()
 40832:20220525:163347.562 In zbx_dshm_create() size:0
 40832:20220525:163347.563 End of zbx_dshm_create():SUCCEED shmid:-1
 40832:20220525:163347.563 End of init_collector_data()
 40832:20220525:163347.563 agent #0 started [main process]
 40833:20220525:163347.563 agent #1 started [collector]
 40833:20220525:163347.564 In init_cpu_collector()
 40833:20220525:163347.564 End of init_cpu_collector():SUCCEED
 40834:20220525:163347.564 agent #2 started [listener #1]
 40834:20220525:163347.564 In zbx_tls_init_child()
 40835:20220525:163347.564 agent #3 started [listener #2]
 40834:20220525:163347.565 GnuTLS library (version 3.7.3) initialized
 40834:20220525:163347.565 End of zbx_tls_init_child()
 40834:20220525:163347.565 zbx_setproctitle() title:'listener #1 [waiting for connection]'
 40836:20220525:163347.565 agent #4 started [listener #3]
 40835:20220525:163347.565 In zbx_tls_init_child()
 40833:20220525:163347.565 zbx_setproctitle() title:'collector [processing data]'
 40836:20220525:163347.565 In zbx_tls_init_child()
 40835:20220525:163347.565 GnuTLS library (version 3.7.3) initialized
 40835:20220525:163347.565 End of zbx_tls_init_child()
 40836:20220525:163347.565 GnuTLS library (version 3.7.3) initialized
 40836:20220525:163347.565 End of zbx_tls_init_child()
 40836:20220525:163347.565 zbx_setproctitle() title:'listener #3 [waiting for connection]'
 40833:20220525:163347.565 In update_cpustats()
 40835:20220525:163347.566 zbx_setproctitle() title:'listener #2 [waiting for connection]'
 40833:20220525:163347.566 End of update_cpustats()
 40833:20220525:163347.566 zbx_setproctitle() title:'collector [idle 1 sec]'
 40836:20220525:163347.862 zbx_setproctitle() title:'listener #3 [processing request]'
 40836:20220525:163347.862 Requested [docker.mem[/rabbitmq,total_cache]]
 40836:20220525:163347.862 In zbx_module_docker_mem()
 40836:20220525:163347.862 docker.mem metrics are not available at the moment - no stat directory
 40836:20220525:163347.862 Sending back [ZBX_NOTSUPPORTED: docker.mem metrics are not available at the moment - no stat directory]
 40836:20220525:163347.862 zbx_setproctitle() title:'listener #3 [waiting for connection]'
 40833:20220525:163348.566 zbx_setproctitle() title:'collector [processing data]'
 40833:20220525:163348.566 In update_cpustats()
 40833:20220525:163348.566 End of update_cpustats()
 40833:20220525:163348.567 zbx_setproctitle() title:'collector [idle 1 sec]'
 40835:20220525:163348.979 zbx_setproctitle() title:'listener #2 [processing request]'
 40835:20220525:163348.979 Requested [docker.mem[/elasticsearch,total_cache]]
 40835:20220525:163348.979 In zbx_module_docker_mem()
 40835:20220525:163348.979 docker.mem metrics are not available at the moment - no stat directory
 40835:20220525:163348.979 Sending back [ZBX_NOTSUPPORTED: docker.mem metrics are not available at the moment - no stat directory]
 40835:20220525:163348.979 zbx_setproctitle() title:'listener #2 [waiting for connection]'
 40834:20220525:163349.205 zbx_setproctitle() title:'listener #1 [processing request]'
 40834:20220525:163349.205 Requested [docker.cpu[/XX-box-cmd-widget,total]]
 40834:20220525:163349.205 In zbx_module_docker_cpu()
 40834:20220525:163349.205 docker.cpu metrics are not available at the moment - no stat directory
 40834:20220525:163349.205 Sending back [ZBX_NOTSUPPORTED: docker.cpu metrics are not available at the moment - no stat directory]
 40834:20220525:163349.205 zbx_setproctitle() title:'listener #1 [waiting for connection]'
 40833:20220525:163349.567 zbx_setproctitle() title:'collector [processing data]'
 40833:20220525:163349.567 In update_cpustats()
 40833:20220525:163349.567 End of update_cpustats()
 40833:20220525:163349.567 zbx_setproctitle() title:'collector [idle 1 sec]'

And when I try to get some value :

:~$ sudo zabbix_agentd -t docker.cpu[/elasticsearch,total]
docker.cpu[/elasticsearch,total]              [m|ZBX_NOTSUPPORTED] [docker.cpu metrics are not available at the moment - no stat directory]

I do have container running:

$ sudo docker ps -a
CONTAINER ID   IMAGE                                         COMMAND                  CREATED       STATUS            PORTS                                       NAMES
aaaaaaaaaaaa   XXXXX/XX-cmd-widget:6.06.707.00.00            "php app/console --e…"   7 hours ago   Up About an hour   9000/tcp                                   XX-box-cmd-widget
aaaaaaaaaaaa   XXXXX/XX-nginx:6.06.707.00.00                 "nginx -g 'daemon of…"   7 hours ago   Up About an hour   0.0.0.0:443->443/tcp                       nginx
aaaaaaaaaaaa   XXXXX/XX-cmd-general:6.06.707.00.00           "php app/console --e…"   7 hours ago   Up About an hour   9000/tcp                                   XX-box-cmd-general
aaaaaaaaaaaa   XXXXX/XX-cmd-XX:6.06.707.00.00                "php app/console --e…"   7 hours ago   Up About an hour   9000/tcp                                   XX-box-cmd-XX
aaaaaaaaaaaa   XXXXX/XX-cmd-elasticsearch:6.06.707.00.00     "php app/console --e…"   7 hours ago   Up About an hour   9000/tcp                                   XX-box-elasticsearch
aaaaaaaaaaaa   XXXXX/XX-php:6.06.707.00.00                   "docker-php-entrypoi…"   7 hours ago   Up About an hour   127.0.0.1:9000->9000/tcp                   XX-box-php
aaaaaaaaaaaa   XXXXX/XX-X:6.06.707.00.00                     "/bin/sh -c 'update-…"   7 hours ago   Up About an hour   22/tcp                                     XX-box-X
aaaaaaaaaaaa   XXXXX/mercure:v0.9.0                          "/mercure"               7 hours ago   Up About an hour   80/tcp, 443/tcp                            mercure-hub
aaaaaaaaaaaa   XXXXX/rabbitmq:3.6                            "docker-entrypoint.s…"   7 hours ago   Up About an hour   4369/tcp, 5671/tcp                         rabbitmq
aaaaaaaaaaaa   XXXXX/elasticsearch:1.7                       "/docker-entrypoint.…"   7 hours ago   Up About an hour   0.0.0.0:9200->9200/tcp,9300/tcp            elasticsearch

I look in this two issues but no luck: https://github.com/monitoringartist/zabbix-docker-monitoring/issues/2 https://github.com/monitoringartist/zabbix-docker-monitoring/issues/117

Do you have some lead for me ? Does your module compatible with Ubuntu 22.04 ?

Best regards

tbjornli commented 1 year ago

I am experiencing the same issue with Zabbix 6.0.10.

i-ky commented 1 year ago

Module opens /proc/mounts file and looks for lines containing cpuset cgroup string: https://github.com/monitoringartist/zabbix-docker-monitoring/blob/f4c80ba11ef00a02f9b3b40fbca10522d9a92e38/src/modules/zabbix_module_docker/zabbix_module_docker.c#L678 For the record, previously it used to be memory cgroup: https://github.com/monitoringartist/zabbix-docker-monitoring/blob/c53765be14eb6b41f820da325879d0c23e46fde0/src/modules/zabbix_module_docker/zabbix_module_docker.c#L678 and /cpuacct cgroup: https://github.com/monitoringartist/zabbix-docker-monitoring/blob/ae6b1f52544f978772b2db5f74febb17b32ee65a/src/modules/zabbix_module_docker/zabbix_module_docker.c#L584

On my Ubuntu 22.04 there is only one line mentioning cgroups:

cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0

and it does not match any of the conditions above.

So, @labanana34, @tbjornli, you can check contents of /proc/mounts on your systems and hack the condition accordingly. But I guess we will need the wisdom of @jangaraj to properly fix the issue without breaking existing setups.

labanana34 commented 7 months ago

Hey there !

For the record, I'm still willing to make the module work on Ubuntu 22.04 But I have no idea about you means with: "you can check contents of /proc/mounts on your systems and hack the condition accordingly" I'll try to take a look.

By the way, is there any reason why the module is not available on Ubuntu 22.04 ?

Best regard.