search

monitoringartist / dockbix-agent-xxl

:whale: Dockerized Zabbix agent with Docker metrics and host metrics support for CoreOS, RHEL, CentOS, Ubuntu, Debian, Fedora, Boot2docker, Photon OS, Amazon Linux, ...
https://hub.docker.com/r/monitoringartist/dockbix-agent-xxl-limited/
Other
182 stars 54 forks source link

No Docker Metrics Available in zabbix #36

Open viveklanjekar opened 6 years ago

viveklanjekar commented 6 years ago

below are info from logs

Docker image: monitoringartist/dockbix-agent-xxl-limited
Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Starting Zabbix Agent [jira02.sg.aws.grabpay.com]. Zabbix 3.4.2 Dockbix Agent XXL (2017-09-26) (revision 72885).
Press Ctrl+C to exit.

     7:20171214:074845.476 Starting Zabbix Agent [jira02.sg.aws.grabpay.com]. Zabbix 3.4.2 Dockbix Agent XXL (2017-09-26) (revision 72885).
     7:20171214:074845.478 **** Enabled features ****
     7:20171214:074845.478 IPv6 support:          YES
     7:20171214:074845.478 TLS support:           YES
     7:20171214:074845.478 **************************
     7:20171214:074845.478 using configuration file: /etc/zabbix/zabbix_agentd.conf
     7:20171214:074845.478 In zbx_load_modules()
     7:20171214:074845.478 loading module "/var/lib/zabbix/modules/zabbix_module_docker.so"
     7:20171214:074845.478 In zbx_module_api_version()
     7:20171214:074845.478 In zbx_module_init()
     7:20171214:074845.478 zabbix_module_docker v0.6.8, compilation time: Sep 26 2017 22:48:15
     7:20171214:074845.478 In zbx_docker_dir_detect()
     7:20171214:074845.479 Cannot detect used docker driver
     7:20171214:074845.479 In zbx_docker_api_detect()
     7:20171214:074845.479 In zbx_module_docker_socket_query()
     7:20171214:074845.479 Docker's socket query: GET /_ping HTTP/1.0
     7:20171214:074845.479 Docker's socket response: OK
     7:20171214:074845.479 Docker's socket works - extended docker metrics are available
     7:20171214:074845.479 In zbx_module_item_list()
     7:20171214:074845.479 In zbx_module_item_timeout()
     7:20171214:074845.479 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_docker.so": /var/lib/zabbix/modules/zabbix_module_docker.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.479 loading module "/var/lib/zabbix/modules/zabbix_module_stress.so"
     7:20171214:074845.479 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_stress.so": /var/lib/zabbix/modules/zabbix_module_stress.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.479 loading module "/var/lib/zabbix/modules/zabbix_module_systemd.so"
     7:20171214:074845.481 [systemd] starting v3.4.2, compiled: Sep 26 2017 22:47:47
     7:20171214:074845.481 [systemd] in cgroup_dir_detect()
     7:20171214:074845.481 [systemd] detected cgroup mount directory: /rootfs/var/lib/docker/overlay2/b3d89b8f3be768a040338eae39ca78d0ae2453df289ad80bac9665b4d761ea61/merged/sys/fs/cgroup/
     7:20171214:074845.481 [systemd] cpu_cgroup is cpuacct
     7:20171214:074845.481 [systemd] set timeout to 3000ms
     7:20171214:074845.481 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_systemd.so": /var/lib/zabbix/modules/zabbix_module_systemd.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.481 loading module "/var/lib/zabbix/modules/zabbix_module_sockets.so"
     7:20171214:074845.481 [sockets] starting v3.4.2
     7:20171214:074845.481 [sockets] set timeout to 3000ms
     7:20171214:074845.481 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_sockets.so": /var/lib/zabbix/modules/zabbix_module_sockets.so: undefined symbol: zbx_module_history_write_cbs
     7:20171214:074845.481 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so, zabbix_module_systemd.so, zabbix_module_sockets.so
     7:20171214:074845.481 End of zbx_load_modules():SUCCEED
     7:20171214:074845.481 In init_collector_data()
     7:20171214:074845.481 In zbx_dshm_create() size:0
     7:20171214:074845.481 End of zbx_dshm_create():SUCCEED shmid:-1
     7:20171214:074845.481 End of init_collector_data()
     7:20171214:074845.481 agent #0 started [main process]
    12:20171214:074845.485 agent #3 started [listener #2]
    11:20171214:074845.485 agent #2 started [listener #1]
    12:20171214:074845.485 In zbx_tls_init_child()
    15:20171214:074845.485 agent #5 started [active checks #1]
    11:20171214:074845.485 In zbx_tls_init_child()
    14:20171214:074845.486 agent #4 started [listener #3]
    15:20171214:074845.486 In zbx_tls_init_child()
    14:20171214:074845.486 In zbx_tls_init_child()
    11:20171214:074845.491 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    10:20171214:074845.503 agent #1 started [collector]
    12:20171214:074845.503 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    10:20171214:074845.503 In init_cpu_collector()
    11:20171214:074845.504 End of zbx_tls_init_child()
    15:20171214:074845.504 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    12:20171214:074845.504 End of zbx_tls_init_child()
    10:20171214:074845.504 End of init_cpu_collector():SUCCEED
    11:20171214:074845.504 __zbx_zbx_setproctitle() title:'listener #1 [waiting for connection]'
    14:20171214:074845.504 OpenSSL library (version OpenSSL 1.0.2g  1 Mar 2016) initialized
    15:2017121

the above logs are from debug level 5

also please find the logs From normal logging 

14:20171214:074414.510 Cannot open metric file: '/rootfs/var/lib/docker/overlay2/c889f774dc895802e1ae039183bbd1d65228868e480e1a23d4a53be89e71ede9/merged/sys/fs/cgroup/memory/1a732c7feb38d73087ab7b2d2f02ce3bcb148e8f2429bd1a6c0c45b1448a8357/memory.stat'`

docker info output

docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.03.2-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.43-17.38.amzn1.x86_64
Operating System: Amazon Linux AMI 2017.03
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.5 MiB
Name: jira02.sg.aws.grabpay.com
ID: L75C:365N:J3CD:RGXY:NOTU:WBAA:SOHM:O5ZT:B7PJ:NDLP:W7UX:YUGO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

OS we are using is AWS linux

further debugging output

[root@jira02 ~]# docker exec -ti dockbix-agent-xxl zabbix_get -s 127.0.0.1 -k docker.mem[/dockbix-agent-xxl,rss]
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "no such file or directory"`

help would be appreciated

jangaraj commented 6 years ago

For the record: some problem zbx_docker_dir_detect():

     7:20171214:074845.478 In zbx_docker_dir_detect()
     7:20171214:074845.479 Cannot detect used docker driver
viveklanjekar commented 6 years ago

any suggestions, what Iam doing wrong? how would I make it work?

jangaraj commented 6 years ago

That's a problem in the module - https://github.com/monitoringartist/zabbix-docker-monitoring/blob/master/src/modules/zabbix_module_docker/zabbix_module_docker.c#L293

viveklanjekar commented 6 years ago

is this something related to permissions?

jangaraj commented 6 years ago

No, the problem is in the source code.

viveklanjekar commented 6 years ago

ok thanks

jangaraj commented 6 years ago

Could you provide outputs of these commands, please?

docker run --rm \
  --net=host \
  --privileged \
  -v /:/rootfs \
  -v /var/run:/var/run \
  -ti ubuntu:16.04 \
  bash -c "cat /proc/mounts | grep 'cpuset cgroup'"

docker run --rm \
  --net=host \
  --privileged \
  -v /:/rootfs \
  -v /var/run:/var/run \
  -ti ubuntu:16.04 \
  bash -c "cat /proc/mounts | grep 'cpuset cgroup' | grep -E '^[^[:blank:]]+[[:blank:]]+(/[^[:blank:]]+/)[^[:blank:]]+[[:blank:]]+cgroup[[:blank:]]+.*$'"
viveklanjekar commented 6 years ago

sorry for late response

below is out of command

[root@jira02 ~]# docker run --rm   --net=host   --privileged   -v /:/rootfs   -v /var/run:/var/run   -ti ubuntu:16.04   bash -c "cat /proc/mounts | grep 'cpuset cgroup'"`
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/overlay2/aab5e773827dd2b811ef4da5c797e3218f8bac0a5b95a633007af1631e9db81d/merged/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0

[root@jira02 ~]# docker run --rm   --net=host   --privileged   -v /:/rootfs   -v /var/run:/var/run   -ti ubuntu:16.04   bash -c "cat /proc/mounts | grep 'cpuset cgroup' | grep -E '^[^[:blank:]]+[[:blank:]]+(/[^[:blank:]]+/)[^[:blank:]]+[[:blank:]]+cgroup[[:blank:]]+.*$'"
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/overlay2/c321de5e9c6d8d2e55477b1f7b4d720e6b1046fb5ea46ad289bc5f00f72cde2c/merged/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
eskogh commented 6 years ago

Any update on this? I got the same error.

Debug output (without --net=host)

root@host1:~# docker run --log-driver json-file --log-opt max-size=500k --name=dockbix-agent-xxl --privileged --restart=unless-stopped --userns=host --hostname=host1 -p 10050:10050 -v /:/rootfs -v /var/run:/var/run -v zabbix-data:/etc/zabbix/zabbix_agentd.d/ --env="ZABBIX_SERVER=10.206.3.98" --env="ZA_ServerActive=10.206.3.98" --env="ZA_Include=/etc/zabbix/zabbix_agentd.d/*.conf" --env="ZA_DebugLevel=5" -d monitoringartist/dockbix-agent-xxl:latest && docker logs -f dockbix-agent-xxl
    10:20180125:143511.481 Starting Zabbix Agent [host1]. Zabbix 3.4.2 Dockbix Agent XXL (2017-10-03) (revision 72885).
    10:20180125:143511.481 **** Enabled features ****
    10:20180125:143511.481 IPv6 support:          YES
    10:20180125:143511.481 TLS support:           YES
    10:20180125:143511.481 **************************
    10:20180125:143511.481 using configuration file: /etc/zabbix/zabbix_agentd.conf
    10:20180125:143511.481 In zbx_load_modules()
    10:20180125:143511.481 loading module "/var/lib/zabbix/modules/zabbix_module_docker.so"
    10:20180125:143511.481 In zbx_module_api_version()
    10:20180125:143511.481 In zbx_module_init()
    10:20180125:143511.481 zabbix_module_docker v0.6.8, compilation time: Oct  3 2017 16:55:38
    10:20180125:143511.481 In zbx_docker_dir_detect()
    10:20180125:143511.482 Detected used docker driver dir: docker/
    10:20180125:143511.482 Detected JoinController cpu,cpuacct
    10:20180125:143511.482 In zbx_docker_api_detect()
    10:20180125:143511.482 In zbx_module_docker_socket_query()
    10:20180125:143511.482 Docker's socket query: GET /_ping HTTP/1.0
    10:20180125:143511.482 Docker's socket response: OK
    10:20180125:143511.482 Docker's socket works - extended docker metrics are available
    10:20180125:143511.482 In zbx_module_item_list()
    10:20180125:143511.482 In zbx_module_item_timeout()
    10:20180125:143511.482 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_docker.so": /var/lib/zabbix/modules/zabbix_module_docker.so: undefined symbol: zbx_module_history_write_cbs
    10:20180125:143511.482 loading module "/var/lib/zabbix/modules/zabbix_module_stress.so"
    10:20180125:143511.482 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_stress.so": /var/lib/zabbix/modules/zabbix_module_stress.so: undefined symbol: zbx_module_history_write_cbs
    10:20180125:143511.482 loading module "/var/lib/zabbix/modules/zabbix_module_systemd.so"
    10:20180125:143511.483 [systemd] starting v3.4.2, compiled: Oct  3 2017 16:55:06
    10:20180125:143511.483 [systemd] in cgroup_dir_detect()
    10:20180125:143511.483 [systemd] detected cgroup mount directory: /rootfs/sys/fs/cgroup/
    10:20180125:143511.483 [systemd] cpu_cgroup is JoinController cpu,cpuacct
    10:20180125:143511.483 [systemd] set timeout to 3000ms
    10:20180125:143511.483 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_systemd.so": /var/lib/zabbix/modules/zabbix_module_systemd.so: undefined symbol: zbx_module_history_write_cbs
    10:20180125:143511.483 loading module "/var/lib/zabbix/modules/zabbix_module_sockets.so"
    10:20180125:143511.483 [sockets] starting v3.4.2
    10:20180125:143511.483 [sockets] set timeout to 3000ms
    10:20180125:143511.483 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_sockets.so": /var/lib/zabbix/modules/zabbix_module_sockets.so: undefined symbol: zbx_module_history_write_cbs
    10:20180125:143511.483 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so, zabbix_module_systemd.so, zabbix_module_sockets.so

Docker info:

root@host1:~# docker info
Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 1.12.6
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 52
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.859 GiB
Name: host1
ID: OY4J:4NWQ:UF7A:5EWP:TGY7:6V3G:RHLC:LA3I:DDEP:3YSG:JNH6:YPPA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: XXXXXXX
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

cat /proc/mounts

root@host1:~# docker run --rm   --net=host   --privileged   -v /:/rootfs   -v /var/run:/var/run   -ti ubuntu:16.04   bash -c "cat /proc/mounts | grep 'cpuset cgroup'"
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/aufs/mnt/5cbb3edc744866e09739f94a39ca84919c415aa2dec4c1c435dadf99c1cb323e/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0

cat /proc/mounts | grep

root@host1:~# docker run --rm   --net=host   --privileged   -v /:/rootfs   -v /var/run:/var/run   -ti ubuntu:16.04   bash -c "cat /proc/mounts | grep 'cpuset cgroup' | grep -E '^[^[:blank:]]+[[:blank:]]+(/[^[:blank:]]+/)[^[:blank:]]+[[:blank:]]+cgroup[[:blank:]]+.*$'"
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/aufs/mnt/2f7d7ba5f2a372dd54631fccaff3f62621c2cbbad291acb3d356321fd8a0bd81/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
viveklanjekar commented 6 years ago

@eskogh for now iam using older version of agent, it works

eskogh commented 6 years ago

@viveklanjekar do you wanna share what version you're using so can I verify that?

viveklanjekar commented 6 years ago

@eskogh use this version docker pull monitoringartist/dockbix-agent-xxl-limited:3.2-2