zabbix / zabbix-docker

Official Zabbix Dockerfiles
https://www.zabbix.com
GNU Affero General Public License v3.0
2.37k stars 1.37k forks source link

Docker: cannot send proxy data to server at x.x.x.x: connection is not allowed #1537

Closed sjackson0109 closed 2 weeks ago

sjackson0109 commented 2 weeks ago
SUMMARY
OS / ENVIRONMENT / Used docker-compose files

OS:

root@gszab01:~/prod# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

Docker:

root@gszab01:~/prod# docker -v
Docker version 27.3.1, build ce12230
CONFIGURATION

ENV:

root@gszab01:~/prod# cat .env
# MySQL settings
MYSQL_DATABASE=zabbix_proxy
MYSQL_USER=zabbix
MYSQL_PASSWORD=<REDACTED>
MYSQL_ROOT_PASSWORD=<REDACTED>
MYSQL_INITDB_SKIP_TZINFO='1'
MYSQL_COLLATION_SERVER=utf8mb4_bin
MYSQL_CHARACTER_SET_SERVER=utf8mb4

# Zabbix Proxy settings
ZBX_DB_SERVER_HOST=zabbix-mysql
ZBX_DB_NAME=zabbix_proxy
ZBX_DB_USER=zabbix
ZBX_DB_PASSWORD=<REDACTED>
ZBX_SERVER_HOST=zabbix.<REDACTED>
ZBX_PROXY_HOSTNAME=gszab01.<REDACTED>

YAML:

root@gszab01:~/prod# cat docker-compose.yaml
services:
  zabbix-mysql:
    image: mysql:8.0
    container_name: zabbix-mysql
    restart: unless-stopped
    environment:
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_INITDB_SKIP_TZINFO: ${MYSQL_INITDB_SKIP_TZINFO}
      MYSQL_COLLATION_SERVER: ${MYSQL_COLLATION_SERVER}
      MYSQL_CHARACTER_SET_SERVER: ${MYSQL_CHARACTER_SET_SERVER}
    volumes:
      - /media/data/mysql:/var/lib/mysql
    networks:
      - zabbix-network

  zabbix-proxy:
    image: zabbix/zabbix-proxy-mysql:alpine-7.0-latest
    container_name: zabbix-proxy
    restart: unless-stopped
    environment:
      DB_SERVER_HOST: ${ZBX_DB_SERVER_HOST}
      MYSQL_DATABASE: ${ZBX_DB_NAME}
      MYSQL_USER: ${ZBX_DB_USER}
      MYSQL_PASSWORD: ${ZBX_DB_PASSWORD}
      ZBX_SERVER_HOST: ${ZBX_SERVER_HOST}
      ZBX_HOSTNAME: ${ZBX_PROXY_HOSTNAME}
    depends_on:
      - zabbix-mysql
    ports:
      - "10051:10051"  # Exposing port 10051 for active proxy communication
      - "10050:10050"  # (Optional) Exposing port 10050 for passive proxy communication
    volumes:
      - /media/data/zabbix-proxy/zabbix_proxy.conf:/etc/zabbix/zabbix_proxy.conf:rw
      - /media/data/zabbix-proxy:/var/lib/zabbix
    networks:
      - zabbix-network

networks:
  zabbix-network:
    driver: bridge
STEPS TO REPRODUCE

Bring it all up:

root@gszab01:~/prod# docker compose down && docker compose up -d --force-recreate --build
[+] Running 3/3
 ✔ Container zabbix-proxy       Removed                                                                                                                                                                                                 2.1s
 ✔ Container zabbix-mysql       Removed                                                                                                                                                                                                 1.6s
 ✔ Network prod_zabbix-network  Removed                                                                                                                                                                                                 0.2s
[+] Running 3/3
 ✔ Network prod_zabbix-network  Created                                                                                                                                                                                                 0.1s
 ✔ Container zabbix-mysql       Started                                                                                                                                                                                                 0.2s
 ✔ Container zabbix-proxy       Started                                                                                                                                                                                                 0.5s
root@gszab01:~/prod# 
EXPECTED RESULTS

Expecting the proxy to successfully handshake with the zabbix server. Note: Firewall rules allows ANY/ANY in both directions between the proxy (10.1.66.0/29) and server (10.101.67.0/29); TCP, UDP and ICMP. These traverse two firewalls, with a site-to-site VPN. I can confirm the VPN passes the traffic, as i can PCAP the traffic on both branch and datacentre firewalls, no drops. I even see the SYN, SYN ACK, SYN PSH, SYN PSH ACK etc... PACKETS ARE DEFINITELY PASSING.

ACTUAL RESULTS

Clearly the handshake packets are passing, but the responses from the Zabbix Server have null length. Note: I have successfully registered 2x other Zabbix Proxies with this server, and this Proxy is a CLONE of a working VM, docker ENV and YAML files included (with volumes and images directories emptied of course).

root@gszab01:~/prod# docker logs zabbix-proxy
** Preparing Zabbix proxy
** Preparing database
** Using MYSQL_USER variable from ENV
** Using MYSQL_PASSWORD variable from ENV
********************
* DB_SERVER_HOST: zabbix-mysql
* DB_SERVER_PORT: 3306
* DB_SERVER_DBNAME: zabbix_proxy
********************
**** MySQL server is not available. Waiting 5 seconds...
** Database 'zabbix_proxy' already exists. Please be careful with database COLLATE!
** Creating 'zabbix_proxy' schema in MySQL
ERROR 1419 (HY000) at line 2493: You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)
** Preparing Zabbix proxy configuration file
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyMode": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "Server": 'zabbix.<REDACTED>'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "Hostname": 'gszab01.<REDACTED>'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "HostnameItem": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ListenIP": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ListenPort": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ListenBacklog": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SourceIP": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "LogType": 'console'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "LogFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "LogFileSize": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "PidFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DebugLevel": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "EnableRemoteCommands": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "LogRemoteCommands": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBHost": 'zabbix-mysql'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBPort": '3306'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBSocket": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBName": 'zabbix_proxy'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBSchema": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "Vault": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VaultDBPath": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VaultTLSCertFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VaultTLSKeyFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VaultPrefix": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VaultURL": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBUser": 'zabbix'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DBPassword": '****'. Enable DEBUG_MODE to view value ...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "AllowUnsupportedDBVersions": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "MaxConcurrentChecksPerPoller": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyBufferMode": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyMemoryBufferAge": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyMemoryBufferSize": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyLocalBuffer": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyOfflineBuffer": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyConfigFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "DataSenderFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StatsAllowedIP": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartPreprocessors": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartAgentPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartIPMIPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartPollersUnreachable": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartTrappers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartPingers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartDiscoverers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartHTTPAgentPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartHTTPPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartODBCPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartSNMPPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "JavaGateway": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "JavaGatewayPort": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartJavaPollers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartVMwareCollectors": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VMwareFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VMwarePerfFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VMwareCacheSize": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "VMwareTimeout": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SNMPTrapperFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartSNMPTrapper": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SocketDir": '/tmp/'...added
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "HousekeepingFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "CacheSize": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartDBSyncers": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "HistoryCacheSize": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "HistoryIndexCacheSize": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "Timeout": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TrapperTimeout": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "UnreachablePeriod": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "UnavailableDelay": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "UnreachableDelay": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "AlertScriptsPath": '/usr/lib/zabbix/alertscripts'...added
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ExternalScripts": '/usr/lib/zabbix/externalscripts'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "FpingLocation": '/usr/sbin/fping'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "Fping6Location": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SSHKeyLocation": '/var/lib/zabbix/ssh_keys'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "LogSlowQueries": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SSLCertLocation": '/var/lib/zabbix/ssl/certs/'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SSLKeyLocation": '/var/lib/zabbix/ssl/keys/'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "SSLCALocation": '/var/lib/zabbix/ssl/ssl_ca/'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "LoadModulePath": '/var/lib/zabbix/modules/'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSConnect": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSAccept": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCAFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCRLFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSServerCertIssuer": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSServerCertSubject": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCertFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCipherAll": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCipherAll13": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCipherCert": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCipherCert13": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCipherPSK": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSCipherPSK13": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSKeyFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSPSKIdentity": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSPSKFile": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "User": 'zabbix'...updated
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "WebDriverURL": ''...removed
** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartBrowserPollers": ''...removed
Starting Zabbix Proxy (active) [gszab01.<REDACTED>]. Zabbix 7.0.4 (revision f383737).

     1:20241009:081032.460 Starting Zabbix Proxy (active) [gszab01.<REDACTED>]. Zabbix 7.0.4 (revision f383737).
     1:20241009:081032.460 **** Enabled features ****
     1:20241009:081032.460 SNMP monitoring:       YES
     1:20241009:081032.460 IPMI monitoring:       YES
     1:20241009:081032.460 Web monitoring:        YES
     1:20241009:081032.460 VMware monitoring:     YES
     1:20241009:081032.460 ODBC:                  YES
     1:20241009:081032.460 SSH support:           YES
     1:20241009:081032.460 IPv6 support:          YES
     1:20241009:081032.460 TLS support:           YES
     1:20241009:081032.460 **************************
     1:20241009:081032.460 using configuration file: /etc/zabbix/zabbix_proxy.conf
     1:20241009:081032.467 Zabbix supports only "utf8_bin,utf8mb3_bin,utf8mb4_bin" collation(s). Database "zabbix_proxy" has default collation "utf8mb4_0900_ai_ci"
     1:20241009:081032.474 character set name or collation name that is not supported by Zabbix found in 579 column(s) of database "zabbix_proxy"
     1:20241009:081032.474 only character set(s) "utf8,utf8mb3,utf8mb4" and corresponding collation(s) "utf8_bin,utf8mb3_bin,utf8mb4_bin" should be used in database
     1:20241009:081032.476 current database version (mandatory/optional): 07000000/07000013
     1:20241009:081032.476 required mandatory version: 07000000
     1:20241009:081032.477 proxy #0 started [main process]
   183:20241009:081032.478 proxy #1 started [configuration syncer #1]
   184:20241009:081032.502 proxy #2 started [trapper #1]
   186:20241009:081032.505 proxy #4 started [trapper #3]
   189:20241009:081032.508 proxy #7 started [preprocessing manager #1]
   188:20241009:081032.508 proxy #6 started [trapper #5]
   190:20241009:081032.508 proxy #8 started [data sender #1]
   185:20241009:081032.511 proxy #3 started [trapper #2]
   187:20241009:081032.511 proxy #5 started [trapper #4]
   191:20241009:081032.512 proxy #9 started [housekeeper #1]
   192:20241009:081032.514 proxy #10 started [http poller #1]
   209:20241009:081032.515 proxy #27 started [unreachable poller #1]
   193:20241009:081032.518 proxy #11 started [browser poller #1]
   214:20241009:081032.518 proxy #32 started [icmp pinger #5]
   215:20241009:081032.518 proxy #33 started [icmp pinger #6]
   213:20241009:081032.518 proxy #31 started [icmp pinger #4]
   212:20241009:081032.519 proxy #30 started [icmp pinger #3]
   211:20241009:081032.520 proxy #29 started [icmp pinger #2]
   183:20241009:081032.521 cannot obtain configuration data from server at "zabbix.<REDACTED>": empty string received
   210:20241009:081032.522 proxy #28 started [icmp pinger #1]
   216:20241009:081032.522 proxy #34 started [icmp pinger #7]
   217:20241009:081032.523 proxy #35 started [icmp pinger #8]
   218:20241009:081032.523 proxy #36 started [icmp pinger #9]
   219:20241009:081032.524 proxy #37 started [icmp pinger #10]
   220:20241009:081032.524 proxy #38 started [icmp pinger #11]
   221:20241009:081032.524 proxy #39 started [icmp pinger #12]
   222:20241009:081032.524 proxy #40 started [icmp pinger #13]
   223:20241009:081032.524 proxy #41 started [icmp pinger #14]
   224:20241009:081032.524 proxy #42 started [icmp pinger #15]
   225:20241009:081032.526 proxy #43 started [icmp pinger #16]
   194:20241009:081032.526 proxy #12 started [discovery manager #1]
   200:20241009:081032.528 proxy #18 started [self-monitoring #1]
   202:20241009:081032.529 proxy #20 started [vmware collector #2]
   203:20241009:081032.529 proxy #21 started [task manager #1]
   204:20241009:081032.529 proxy #22 started [poller #1]
   205:20241009:081032.529 proxy #23 started [poller #2]
   206:20241009:081032.530 proxy #24 started [poller #3]
   195:20241009:081032.530 proxy #13 started [history syncer #1]
   208:20241009:081032.531 proxy #26 started [poller #5]
   201:20241009:081032.531 proxy #19 started [vmware collector #1]
   196:20241009:081032.531 proxy #14 started [history syncer #2]
   207:20241009:081032.532 proxy #25 started [poller #4]
   198:20241009:081032.532 proxy #16 started [history syncer #4]
   197:20241009:081032.532 proxy #15 started [history syncer #3]
   199:20241009:081032.533 proxy #17 started [snmp trapper #1]
   227:20241009:081032.540 proxy #45 started [icmp pinger #18]
   226:20241009:081032.540 proxy #44 started [icmp pinger #17]
   230:20241009:081032.541 proxy #48 started [icmp pinger #21]
   229:20241009:081032.542 proxy #47 started [icmp pinger #20]
   228:20241009:081032.542 proxy #46 started [icmp pinger #19]
   231:20241009:081032.543 proxy #49 started [icmp pinger #22]
   232:20241009:081032.543 proxy #50 started [icmp pinger #23]
   233:20241009:081032.544 proxy #51 started [icmp pinger #24]
   235:20241009:081032.544 proxy #53 started [icmp pinger #26]
   236:20241009:081032.544 proxy #54 started [icmp pinger #27]
   237:20241009:081032.544 proxy #55 started [icmp pinger #28]
   234:20241009:081032.545 proxy #52 started [icmp pinger #25]
   238:20241009:081032.545 proxy #56 started [icmp pinger #29]
   239:20241009:081032.546 proxy #57 started [icmp pinger #30]
   243:20241009:081032.560 proxy #61 started [icmp pinger #34]
   242:20241009:081032.561 proxy #60 started [icmp pinger #33]
   244:20241009:081032.561 proxy #62 started [icmp pinger #35]
   246:20241009:081032.561 proxy #64 started [icmp pinger #37]
   248:20241009:081032.561 proxy #66 started [icmp pinger #39]
   249:20241009:081032.562 proxy #67 started [icmp pinger #40]
   247:20241009:081032.562 proxy #65 started [icmp pinger #38]
   245:20241009:081032.562 proxy #63 started [icmp pinger #36]
   250:20241009:081032.562 proxy #68 started [icmp pinger #41]
   241:20241009:081032.562 proxy #59 started [icmp pinger #32]
   251:20241009:081032.564 proxy #69 started [icmp pinger #42]
   252:20241009:081032.564 proxy #70 started [icmp pinger #43]
   253:20241009:081032.564 proxy #71 started [icmp pinger #44]
   240:20241009:081032.565 proxy #58 started [icmp pinger #31]
   254:20241009:081032.565 proxy #72 started [icmp pinger #45]
   255:20241009:081032.565 proxy #73 started [icmp pinger #46]
   256:20241009:081032.566 proxy #74 started [icmp pinger #47]
   257:20241009:081032.566 proxy #75 started [icmp pinger #48]
   258:20241009:081032.571 proxy #76 started [icmp pinger #49]
   259:20241009:081032.572 proxy #77 started [icmp pinger #50]
   260:20241009:081032.572 proxy #78 started [availability manager #1]
   265:20241009:081032.574 proxy #83 started [internal poller #1]
   264:20241009:081032.576 proxy #82 started [snmp poller #1]
   263:20241009:081032.576 proxy #81 started [agent poller #1]
   261:20241009:081032.577 proxy #79 started [odbc poller #1]
   262:20241009:081032.579 proxy #80 started [http agent poller #1]
   264:20241009:081032.581 thread started
   263:20241009:081032.583 thread started
   262:20241009:081032.584 thread started
   189:20241009:081032.584 [3] thread started [preprocessing worker #3]
   189:20241009:081032.584 [1] thread started [preprocessing worker #1]
   189:20241009:081032.587 [2] thread started [preprocessing worker #2]
   194:20241009:081032.591 thread started [discovery worker #1]
   194:20241009:081032.592 thread started [discovery worker #2]
   194:20241009:081032.592 thread started [discovery worker #3]
   190:20241009:081032.676 cannot send proxy data to server at "zabbix.<REDACTED>": connection is not allowed
   190:20241009:081033.690 cannot send proxy data to server at "zabbix.<REDACTED>": connection is not allowed
   190:20241009:081034.703 cannot send proxy data to server at "zabbix.<REDACTED>": connection is not allowed
   190:20241009:081035.724 cannot send proxy data to server at "zabbix.<REDACTED>": connection is not allowed
   190:20241009:081036.737 cannot send proxy data to server at "zabbix.<REDACTED>": connection is not allowed
VERIFYING PORTS ARE OPEN

We can verify the tcp sockets open, bt sending a network connection SYN request to the IP/port, from within the docker zabbix-proxy instance:

root@gszab01:~/prod# docker ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED          STATUS         PORTS                                                                   NAMES
966f8beafd26   zabbix/zabbix-proxy-mysql:alpine-7.0-latest   "/usr/bin/docker-ent…"   9 minutes ago    Up 9 minutes   0.0.0.0:10050-10051->10050-10051/tcp, :::10050-10051->10050-10051/tcp   zabbix-proxy
b03d87e76426   mysql:8.0                                     "docker-entrypoint.s…"   10 minutes ago   Up 9 minutes   3306/tcp, 33060/tcp                                                     zabbix-mysql
root@gszab01:~/prod# docker exec -it zabbix-proxy bash
966f8beafd26:/var/lib/zabbix$ nc -zv zabbix.<REDACTED> 10051
zabbix.<REDACTED> (10.101.67.2:10051) open
966f8beafd26:/var/lib/zabbix$

Aside from me removing the FQDN, no other changes to the logs/output has been modified.

Proxy on the zabbix frontend is configured with: PROXY / Name: gszab01. PROXY / Name: null PROXY / Mode: Active PROXY / Address: gszab01. ENCRYPTION: Connections to Proxy: no encryption ENCRYPTION: Connections from proxy: no encryption checked Timeouts: All defaults.

sjackson0109 commented 2 weeks ago

I'm going to tackle the collation separately. but i don't feel the DB collation is related to the proxy<>server communication.

Any advice?

dotneft commented 2 weeks ago

Looks like the proxy's IP is not allowed for connections to Zabbix server according configuration of proxy on Zabbix server side.

sjackson0109 commented 2 weeks ago

Log from the docker Zabbix server:

docker logs prod-zabbix-server-1
   262:20241009:084450.376 cannot accept connection from proxy "gszab01.<REDACTED>" at "10.1.66.2", allowed address: "gszab01.<REDACTED>": connection is not allowed
   265:20241009:084450.564 cannot accept connection from proxy "gszab01.<REDACTED>" at "10.1.66.2", allowed address: "gszab01.<REDACTED>": connection is not allowed
   263:20241009:084451.577 cannot accept connection from proxy "gszab01.<REDACTED>" at "10.1.66.2", allowed address: "gszab01.<REDACTED>": connection is not allowed
   266:20241009:084452.591 cannot accept connection from proxy "gszab01.<REDACTED>" at "10.1.66.2", allowed address: "gszab01.<REDACTED>": connection is not allowed
   265:20241009:084453.605 cannot accept connection from proxy "gszab01.<REDACTED>" at "10.1.66.2", allowed address: "gszab01.<REDACTED>": connection is not allowed
   265:20241009:084454.618 cannot accept connection from proxy "gszab01.<REDACTED>" at "10.1.66.2", allowed address: "gszab01.<REDACTED>": connection is not allowed

Confirming the precise zabbix_proxy.conf parameters, which i missed earlier:

root@gszab01:~/prod# cat /media/data/zabbix-proxy/zabbix_proxy.conf | grep -E '^Server|^ServerActive|^Hostname'
Server=zabbix.<REDACTED>
Hostname=gszab01.<REDACTED>
root@gszab01:~/prod#
sjackson0109 commented 2 weeks ago

Found it!

FIX: Zabbix Server frontend> Administration > Proxies > Click to EDIT the proxy, and clear the proxy address attribute.