search

zabbix / zabbix-docker

Official Zabbix Dockerfiles
https://www.zabbix.com
GNU Affero General Public License v3.0
2.39k stars 1.38k forks source link

Is it possible to use the docker zabbix-server with external zabbix-agent ? #1097

Closed hallux4 closed 1 year ago

hallux4 commented 1 year ago
SUMMARY

Hello

We currently have 2 zabbix instances.

OLD_Zabbix_server => systemd 4.4 NEW_Zabbix_server => docker 6.4

I try to migrate our OLD_Zabbix_server from a VM with systemd services toward a NEW_Zabbix_server running on Docker. Database has been copied and upgraded to avoid redeclaring all the hosts and configuration.

When I use the zabbix-agent2 container, I can see the Zabbix server availability (ZBX status green <=> zabbix-agent:10050). None of the other hosts zabbix-agents seems to work properly.

If I shutdown the zabbix-docker-agent2 on my NEW_Zabbix_server host and start a zabbix service with systemd. I replaced zabbix-agent:dns with NEW_Zabbix_server:ip In the web interface I get this message.

NEW_Zabbix_server:10050 
Not available 
Get value from agent failed: cannot connect to [[NEW_Zabbix_server]:10050]: [4] Interrupted system call

no firewall rules blocking the way.

# from the host
telnet NEW_Zabbix_server 10050
Trying NEW_Zabbix_server...
Connected to NEW_Zabbix_server.
Escape character is '^]'.
Connection closed by foreign host.
OS / ENVIRONMENT / Used docker-compose files

NEW_Zabbix_server : Debian 12.1 Agent: zabbix_agent2 (Zabbix) 6.4.6 - Revision 52b7e2cb506 22 August 2023 image: zabbix/zabbix-server-mysql:ubuntu-6.4-latest image: zabbix/zabbix-web-apache-mysql:ubuntu-6.4-latest image: zabbix/zabbix-java-gateway:ubuntu-6.4-latest image: zabbix/zabbix-snmptraps:ubuntu-6.4-latest image: mysql:8.0

CONFIGURATION

zabbix-agent2 conf

egrep -v '#|(^$)' /etc/zabbix/zabbix_agent2.conf
PidFile=/var/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=0
DebugLevel=4
Server=OLD_Zabbix_server,NEW_Zabbix_server
ServerActive=OLD_Zabbix_server,NEW_Zabbix_server
Hostname=zabbix-srv-core
Include=/etc/zabbix/zabbix_agent2.d/*.conf
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
Include=./zabbix_agent2.d/plugins.d/*.conf

Docker compose

version: '3.5'
services:
 zabbix-server:
  image: zabbix/zabbix-server-mysql:ubuntu-6.4-latest
  ports:
   - "10051:10051"
  volumes:
   - /etc/localtime:/etc/localtime:ro
   - ./zbx_env/usr/lib/zabbix/alertscripts:/usr/lib/zabbix/alertscripts:ro
   - ./zbx_env/usr/lib/zabbix/externalscripts:/usr/lib/zabbix/externalscripts:ro
   - ./zbx_env/var/lib/zabbix/dbscripts:/var/lib/zabbix/dbscripts:ro
   - ./zbx_env/var/lib/zabbix/export:/var/lib/zabbix/export:rw
   - ./zbx_env/var/lib/zabbix/modules:/var/lib/zabbix/modules:ro
   - ./zbx_env/var/lib/zabbix/enc:/var/lib/zabbix/enc:ro
   - ./zbx_env/var/lib/zabbix/ssh_keys:/var/lib/zabbix/ssh_keys:ro
   - ./zbx_env/var/lib/zabbix/mibs:/var/lib/zabbix/mibs:ro
   - snmptraps:/var/lib/zabbix/snmptraps:rw
  ulimits:
   nproc: 65535
   nofile:
    soft: 20000
    hard: 40000
  env_file:
   - ./env_vars/.env_db_mysql
   - ./env_vars/.env_srv
  secrets:
   - MYSQL_USER
   - MYSQL_PASSWORD
   - MYSQL_ROOT_USER
   - MYSQL_ROOT_PASSWORD
  depends_on:
   - mysql-server
  networks:
   zbx_net_backend:
     aliases:
      - zabbix-server
      - zabbix-server-mysql
      - zabbix-server-ubuntu-mysql
      - zabbix-server-mysql-ubuntu
   zbx_net_frontend:
#  devices:
#   - "/dev/ttyUSB0:/dev/ttyUSB0"
  stop_grace_period: 30s
  sysctls:
   - net.ipv4.ip_local_port_range=1024 64999
   - net.ipv4.conf.all.accept_redirects=0
   - net.ipv4.conf.all.secure_redirects=0
   - net.ipv4.conf.all.send_redirects=0
  labels:
   com.zabbix.description: "Zabbix server with MySQL database support"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "zabbix-server"
   com.zabbix.dbtype: "mysql"
   com.zabbix.os: "ubuntu"

 zabbix-web-apache-mysql:
  image: zabbix/zabbix-web-apache-mysql:ubuntu-6.4-latest
  ports:
   - "8081:8080"
   - "8443:8443"
  volumes:
   - /etc/localtime:/etc/localtime:ro
   - ./zbx_env/etc/ssl/apache2:/etc/ssl/apache2:ro
  env_file:
   - ./env_vars/.env_db_mysql
   - ./env_vars/.env_web
  secrets:
   - MYSQL_USER
   - MYSQL_PASSWORD
  depends_on:
   - mysql-server
   - zabbix-server
  healthcheck:
   test: ["CMD", "curl", "-f", "http://localhost:8080/"]
   interval: 10s
   timeout: 5s
   retries: 3
   start_period: 30s
  networks:
   zbx_net_backend:
    aliases:
     - zabbix-web-apache-mysql
     - zabbix-web-apache-ubuntu-mysql
     - zabbix-web-apache-mysql-ubuntu
   zbx_net_frontend:
  stop_grace_period: 10s
  sysctls:
   - net.core.somaxconn=65535
  labels:
   com.zabbix.description: "Zabbix frontend on Apache web-server with MySQL database support"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "zabbix-frontend"
   com.zabbix.webserver: "apache2"
   com.zabbix.dbtype: "mysql"
   com.zabbix.os: "ubuntu"

# zabbix-agent:
#  image: zabbix/zabbix-agent2:ubuntu-6.4-latest
#  ports:
#   - "10050:10050"
#  volumes:
#   - /etc/localtime:/etc/localtime:ro
#   - ./zbx_env/etc/zabbix/zabbix_agentd.d:/etc/zabbix/zabbix_agentd.d:ro
#   - ./zbx_env/var/lib/zabbix/modules:/var/lib/zabbix/modules:ro
#   - ./zbx_env/var/lib/zabbix/enc:/var/lib/zabbix/enc:ro
#   - ./zbx_env/var/lib/zabbix/ssh_keys:/var/lib/zabbix/ssh_keys:ro
#  env_file:
#   - ./env_vars/.env_agent
#  privileged: true
#  pid: "host"
#  networks:
#   zbx_net_backend:
#    aliases:
#     - zabbix-agent
#     - zabbix-agent-passive
#     - zabbix-agent-ubuntu
#  stop_grace_period: 5s
#  labels:
#   com.zabbix.description: "Zabbix agent"
#   com.zabbix.company: "Zabbix LLC"
#   com.zabbix.component: "zabbix-agentd"
#   com.zabbix.os: "ubuntu"

 zabbix-java-gateway:
  image: zabbix/zabbix-java-gateway:ubuntu-6.4-latest
  ports:
   - "10052:10052"
  env_file:
   - ./env_vars/.env_java
  networks:
   zbx_net_backend:
    aliases:
     - zabbix-java-gateway
     - zabbix-java-gateway-ubuntu
  stop_grace_period: 5s
  labels:
   com.zabbix.description: "Zabbix Java Gateway"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "java-gateway"
   com.zabbix.os: "ubuntu"

 zabbix-snmptraps:
  image: zabbix/zabbix-snmptraps:ubuntu-6.4-latest
  ports:
   - "162:1162/udp"
  volumes:
   - snmptraps:/var/lib/zabbix/snmptraps:rw
  networks:
   zbx_net_frontend:
    aliases:
     - zabbix-snmptraps
   zbx_net_backend:
  stop_grace_period: 5s
  labels:
   com.zabbix.description: "Zabbix snmptraps"
   com.zabbix.company: "Zabbix LLC"
   com.zabbix.component: "snmptraps"
   com.zabbix.os: "ubuntu"

 mysql-server:
  image: mysql:8.0
  command:
   - mysqld
   - --character-set-server=utf8mb4
   - --collation-server=utf8mb4_bin
   - --log_bin_trust_function_creators=1
   - --default-authentication-plugin=mysql_native_password
  volumes:
   - ./zbx_env/var/lib/mysql:/var/lib/mysql:rw
  env_file:
   - ./env_vars/.env_db_mysql
  secrets:
   - MYSQL_USER
   - MYSQL_PASSWORD
   - MYSQL_ROOT_PASSWORD
  stop_grace_period: 1m
  networks:
   zbx_net_backend:
    aliases:
     - mysql-server
     - zabbix-database
     - mysql-database

# db_data_mysql:
#  image: busybox
#  volumes:
#   - ./zbx_env/var/lib/mysql:/var/lib/mysql:rw

# elasticsearch:
#  image: elasticsearch
#  profiles:
#   - full
#   - all
#  environment:
#   - transport.host=0.0.0.0
#   - discovery.zen.minimum_master_nodes=1
#  networks:
#   zbx_net_backend:
#    aliases:
#     - elasticsearch

networks:
  zbx_net_frontend:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
    ipam:
      driver: default
      config:
      - subnet: 172.16.238.0/24
  zbx_net_backend:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
    #internal: true
    ipam:
      driver: default
      config:
      - subnet: 172.16.239.0/24

volumes:
  snmptraps:

secrets:
  MYSQL_USER:
    file: ./env_vars/.MYSQL_USER
  MYSQL_PASSWORD:
    file: ./env_vars/.MYSQL_PASSWORD
  MYSQL_ROOT_USER:
    file: ./env_vars/.MYSQL_ROOT_USER
  MYSQL_ROOT_PASSWORD:
    file: ./env_vars/.MYSQL_ROOT_PASSWORD

env_vars

env_vars/.env_agent:ZBX_HOSTNAME=zabbix-srv-core
env_vars/.env_srv:ZBX_JAVAGATEWAY_ENABLE=true
env_vars/.env_srv:ZBX_STARTJAVAPOLLERS=5
env_vars/.env_srv:ZBX_ENABLE_SNMP_TRAPS=true
env_vars/.env_web:ZBX_SERVER_HOST=zabbix-server
env_vars/.env_web:ZBX_SERVER_NAME=Composed installation

In the NEW_Zabbix_server log file /var/log/zabbix/zabbix_agent2.log OLD_Zabbix_server get updates

2023/09/19 01:38:11.047852 sending passive check response: '138152419328' to 'OLD_Zabbix_server '
2023/09/19 01:38:11.208306 received passive check request: 'vfs.fs.size[/boot,used]' from 'OLD_Zabbix_server '
2023/09/19 01:38:11.208455 [1] processing update request (1 requests)

NEW_Zabbix_server get nothing

2023/09/19 01:38:14.514932 [102] In refreshActiveChecks() from [NEW_Zabbix_server:10051 ]
2023/09/19 01:38:14.516679 [102] End of refreshActiveChecks() from [NEW_Zabbix_server:10051 ]
2023/09/19 01:38:15.524379 [101] processing update request (0 requests)
2023/09/19 01:38:15.524385 [101] skipping empty update for unregistered client

I also tried all of the above with 172.16.239.1 (host gateway on backend network).

Same result.

I don' t understand why only the OLD_Zabbix_server get a green status on ZBX availability while interrogating the NEW_Zabbix_server' s agent.

I'm very new to Zabbix, and not very experienced with Docker. Is there something obvious that I do wrong ?

Any help is welcome. Thanks you.

dotneft commented 1 year ago

Zabbix server does not use 10050 port, it uses 10051 port. Only Zabbix agent uses 10050 port.

hallux4 commented 1 year ago

Sorry if I wasn't clear. When I wrote "server", I was referring to the VM that host the zabbix docker container and the zabbix-agent2.service (systemd).

https://www.zabbix.com/documentation/current/en/manual/quickstart/host

agent

The agent that I define for my new host (new zabbix server) can't connect properly (from the container to the host). the connection works from the old zabbix server though (same agent, same config, different VM).

dotneft commented 1 year ago

Firewall? Check telnet / nc.

hallux4 commented 1 year ago

it was indeed a firewall problem.

I restarted the zabbix-server container as root for debugging.

apt update
apt install netcat tcpdump telnet net-tools

# on the host :
ufw disable
nc -l -vv -p 5000
Listening on 0.0.0.0 5000
...
# on the container:
nc NEW_Zabbix_server 5000

# it then said :
Connection received on 172.16.239.5 4772
ufw enable
ufw allow from 172.16.239.0/24

I naively believed that the docker container outgoing connection directed to the host would be considered as "local connection" :'(

after updating the zabbix-agent2.conf

Server=172.16.239.5,OLD_Zabbix_server #passive checks
ServerActive=NEW_Zabbix_server,OLD_Zabbix_server  #active checks
...
systemctl restart zabbix-agent2.service

tail -f

Everything worked a lot better.

thanks for your help.

dotneft commented 1 year ago

so the problem is fixed? :-)

hallux4 commented 1 year ago

everything is ok, thanks