search

zabbix / zabbix-docker

Official Zabbix Dockerfiles
https://www.zabbix.com
GNU Affero General Public License v3.0
2.35k stars 1.37k forks source link

docker compose upgrade gone bad resulting in HTTP 500 #1448

Closed bertalanimre closed 3 months ago

bertalanimre commented 3 months ago
SUMMARY

After pulling latest Zabbix images (previously on 6.4), the WebUI responds with HTTP 500 and complains about the PHP version. Did I forget to update anything else other than just pulling in the latest images? I haven't found anything specific task in the upgrade procedures I should be doing, tho to be fair, it is not about docker compose running environment at all so I am probably be at fault here.

OS / ENVIRONMENT / Used docker-compose files

Ubuntu 22.04 Docker version 27.0.3, build 7d4bcd8 Docker Compose version v2.28.1

CONFIGURATION

docker-compose.yml

version: '3.8'
services:
  zabbix-server:
    image: zabbix/zabbix-server-mysql:latest
    container_name: zabbix-server
    ports:
      - '10051:10051'
    environment:
      DB_SERVER_HOST: XXX.XXX.XXX.XXX
      DB_SERVER_PORT: 3306
      MYSQL_USER: topsecretuser
      MYSQL_PASSWORD: topsecretpassword
      MYSQL_DATABASE: zabbix
    volumes:
      - './alertscripts:/usr/lib/zabbix/alertscripts'
      - './zabbix_server.conf:/etc/zabbix/zabbix_server.conf'
  zabbix-proxy:
    image: zabbix/zabbix-proxy-mysql:latest
    container_name: zabbix-proxy
    ports:
      - '10050:10050'
    environment:
      DB_SERVER_HOST: XXX.XXX.XXX.XXX
      DB_SERVER_PORT: 3306
      MYSQL_USER: topsecretuser
      MYSQL_PASSWORD: topsecretpassword
      MYSQL_DATABASE: zabbix
      ZBX_SERVER_HOST:  XXX.XXX.XXX.XXX
      ZBX_SERVER_PORT: 10051
      ZBX_SERVER_NAME: "MyCompany Zabbix"
  zabbix-web:
    image: zabbix/zabbix-web-nginx-mysql:latest
    container_name: zabbix-web
    depends_on:
      - "zabbix-server"
    ports:
      - '80:8080'
      - '443:8443'
    environment:
      DB_SERVER_HOST: XXX.XXX.XXX.XXX
      DB_SERVER_PORT: 3306
      MYSQL_USER: topsecretuser
      MYSQL_PASSWORD: topsecretpassword
      MYSQL_DATABASE: zabbix
      PHP_TZ: Europe/Budapest
      ZBX_SERVER_HOST:  XXX.XXX.XXX.XXX
      ZBX_SERVER_PORT: 10051
      ZBX_SERVER_NAME: "MyCompany Zabbix"
    volumes:
      - '../ssl:/etc/ssl/nginx'
      - './zabbix-sql-scripts:/usr/share/zabbix/zabbix-sql-scripts'
      - './CUser.php:/usr/share/zabbix/include/classes/api/services/CUser.php'

Inside the container

myuser@zabbix-server:/opt/zabbix-docker# docker exec -it <container_id> bash
<container_id>:/usr/share/zabbix$ php-fpm83 -v
PHP 8.3.8 (fpm-fcgi) (built: Jun  6 2024 16:52:05)
Copyright (c) The PHP Group
Zend Engine v4.3.8, Copyright (c) Zend Technologies
<container_id>:/etc/nginx/http.d$ ls -lah /tmp/php-fpm.sock
srw-rw----    1 zabbix   zabbix         0 Jul  4 11:52 /tmp/php-fpm.sock
<container_id>:/var/log$ ps aux | grep php
   19 zabbix    0:00 {php-fpm83} php-fpm: master process (/etc/php83/php-fpm.conf)
   25 zabbix    0:00 {php-fpm83} php-fpm: pool zabbix
   26 zabbix    0:00 {php-fpm83} php-fpm: pool zabbix
   27 zabbix    0:00 {php-fpm83} php-fpm: pool zabbix
   28 zabbix    0:00 {php-fpm83} php-fpm: pool zabbix
   29 zabbix    0:00 {php-fpm83} php-fpm: pool zabbix
   46 zabbix    0:00 {php-fpm83} php-fpm: pool zabbix
   64 zabbix    0:00 grep php
STEPS TO REPRODUCE

I don't know who to exaclty reproduce the issue, but what I've done in order to upgrade to the lastest docker images were:

docker compose stop
docker compose rm -f
docker compose pull
docker compose up -d

I have not changed any configuration line or executed any command before or after the docker image changes

EXPECTED RESULTS

I honestly just expected the WebUI to load in

ACTUAL RESULTS

The website comes back with a browser supplied HTTP 500. Uplon calling the URL, the following lines are coming to the docker compose logs: (I've hid the IP addresses and URLs, they are normal)

zabbix-web     | ** Deploying Zabbix web-interface (Nginx) with MySQL database
zabbix-web     | ** Using MYSQL_USER variable from ENV
zabbix-web     | ** Using MYSQL_PASSWORD variable from ENV
zabbix-web     | ********************
zabbix-web     | * DB_SERVER_HOST: XXX:XXX.XXX.XXX
zabbix-web     | * DB_SERVER_PORT: 3306
zabbix-web     | * DB_SERVER_DBNAME: zabbix
zabbix-web     | ********************
zabbix-web     | ** Adding Zabbix virtual host (HTTP)
zabbix-web     | ** Enable SSL support for Nginx
zabbix-web     | ** Preparing Zabbix frontend configuration file
zabbix-web     | ########################################################
zabbix-web     | ** Executing supervisord
zabbix-web     | 2024-07-04 12:40:40,447 INFO Included extra file "/etc/supervisor/conf.d/supervisord_zabbix.conf" during parsing
zabbix-web     | 2024-07-04 12:40:40,447 INFO Included extra file "/etc/supervisor/conf.d/supervisord_zabbix.conf" during parsing
zabbix-web     | 2024-07-04 12:40:40,464 INFO RPC interface 'supervisor' initialized
zabbix-web     | 2024-07-04 12:40:40,464 INFO RPC interface 'supervisor' initialized
zabbix-web     | 2024-07-04 12:40:40,464 INFO supervisord started with pid 1
zabbix-web     | 2024-07-04 12:40:40,464 INFO supervisord started with pid 1
zabbix-web     | 2024-07-04 12:40:41,468 INFO spawned: 'nginx' with pid 18
zabbix-web     | 2024-07-04 12:40:41,468 INFO spawned: 'nginx' with pid 18
zabbix-web     | 2024-07-04 12:40:41,476 INFO spawned: 'php-fpm83' with pid 19
zabbix-web     | 2024-07-04 12:40:41,476 INFO spawned: 'php-fpm83' with pid 19
zabbix-web     | 2024/07/04 12:40:41 [warn] 18#18: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/http.d/nginx_ssl.conf:2
zabbix-web     | nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/http.d/nginx_ssl.conf:2
zabbix-web     | 2024/07/04 12:40:41 [warn] 18#18: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/http.d/nginx_ssl.conf:3
zabbix-web     | nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/http.d/nginx_ssl.conf:3
zabbix-web     | [04-Jul-2024 12:40:41] NOTICE: fpm is running, pid 19
zabbix-web     | [04-Jul-2024 12:40:41] NOTICE: ready to handle connections
zabbix-web     | 2024-07-04 12:40:43,551 INFO success: nginx entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
zabbix-web     | 2024-07-04 12:40:43,551 INFO success: nginx entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
zabbix-web     | 2024-07-04 12:40:43,552 INFO success: php-fpm83 entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
zabbix-web     | 2024-07-04 12:40:43,552 INFO success: php-fpm83 entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
.....
zabbix-web     | 2024/07/04 11:54:45 [error] 22#22: *3 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined method CSettingsHelper::getGlobal() in /usr/share/zabbix/include/classes/api/services/CUser.php:2375
zabbix-web     | Stack trace:
zabbix-web     | #0 /usr/share/zabbix/include/classes/api/services/CUser.php(1723): CUser->addExtraFields()
zabbix-web     | #1 /usr/share/zabbix/include/classes/api/clients/CLocalApiClient.php(126): CUser->checkAuthentication()
zabbix-web     | #2 /usr/share/zabbix/include/classes/api/wrappers/CFrontendApiWrapper.php(94): CLocalApiClient->callMethod()
zabbix-web     | #3 /usr/share/zabbix/include/classes/api/wrappers/CApiWrapper.php(89): CFrontendApiWrapper->callClientMethod()
zabbix-web     | #4 /usr/share/zabbix/include/classes/api/wrappers/CFrontendApiWrapper.php(58): CApiWrapper->callMethod()
zabbix-web     | #5 /usr/share/zabbix/include/classes/api/wrappers/CApiWrapper.php(77): CFrontendApiWrapper->callMethod()
zabbix-web     | #6 /usr/share/zabbix/include/classes/user/CWebUser.php(94): CApiWrapper->__call()
zabbix-web     | #7 /usr/share/zabbix/include/classes/core/ZBase.php(550): CWebUser::checkAuthentication()
zabbix-web     | #8 /usr/share/zabbix/include/classes" while reading response header from upstream, client: YYY.YYY.YYY.YYY, server: zabbix, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/tmp/php-fpm.sock:", host: "zabbix.mycompany.com"
zabbix-web     | CCC.CCC.CCC.CCC - - [04/Jul/2024:11:54:45 +0000] "GET / HTTP/2.0" 500 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "85.238.75.132"
zabbix-server  |    216:20240704:115447.320 cannot send list of active checks to "AAA.AAA.AAA.AAA": host [MyServer1] not found
zabbix-server  |    213:20240704:115448.193 cannot send list of active checks to "BBB.BBB.BBB.BBB": host [MyServer2] not found
zabbix-server  |    190:20240704:115448.195 [3] cannot convert value from double to uint64
zabbix-server  |    190:20240704:115449.191 [1] cannot convert value from double to uint64

Oh wait, I might have found something. Turn out, the Proxy server dies during "boot".

...
zabbix-proxy   | ** Preparing Zabbix proxy
zabbix-proxy   | ** Preparing database
zabbix-proxy   | ** Using MYSQL_USER variable from ENV
zabbix-proxy   | ** Using MYSQL_PASSWORD variable from ENV
zabbix-proxy   | ********************
zabbix-proxy   | * DB_SERVER_HOST: XXX.XXX.XXX.XXX
zabbix-proxy   | * DB_SERVER_PORT: 3306
zabbix-proxy   | * DB_SERVER_DBNAME: zabbix
zabbix-proxy   | ********************
zabbix-proxy   | ** Database 'zabbix' already exists. Please be careful with database COLLATE!
zabbix-proxy   | ** Table 'zabbix.dbversion' already exists.
zabbix-proxy   | ** Preparing Zabbix proxy configuration file
zabbix-proxy   | ** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "ProxyMode": ''...removed
....
zabbix-proxy   | ** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "TLSPSKFile": ''...removed
zabbix-proxy   | ** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "User": 'zabbix'...added
zabbix-proxy   | ** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "WebDriverURL": ''...removed
zabbix-proxy   | ** Updating '/etc/zabbix/zabbix_proxy.conf' parameter "StartBrowserPollers": ''...removed
zabbix-proxy   | Starting Zabbix Proxy (active) [zabbix-proxy-mysql]. Zabbix 7.0.0 (revision 49955f1).
zabbix-proxy   | Press Ctrl+C to exit.
zabbix-proxy   | 
zabbix-proxy   |      1:20240704:124040.401 Starting Zabbix Proxy (active) [zabbix-proxy-mysql]. Zabbix 7.0.0 (revision 49955f1).
zabbix-proxy   |      1:20240704:124040.402 **** Enabled features ****
zabbix-proxy   |      1:20240704:124040.403 SNMP monitoring:       YES
zabbix-proxy   |      1:20240704:124040.404 IPMI monitoring:       YES
zabbix-proxy   |      1:20240704:124040.404 Web monitoring:        YES
zabbix-proxy   |      1:20240704:124040.404 VMware monitoring:     YES
zabbix-proxy   |      1:20240704:124040.405 ODBC:                  YES
zabbix-proxy   |      1:20240704:124040.406 SSH support:           YES
zabbix-proxy   |      1:20240704:124040.406 IPv6 support:          YES
zabbix-proxy   |      1:20240704:124040.406 TLS support:           YES
zabbix-proxy   |      1:20240704:124040.406 **************************
zabbix-proxy   |      1:20240704:124040.407 using configuration file: /etc/zabbix/zabbix_proxy.conf
zabbix-proxy   |      1:20240704:124040.415 cannot use database "zabbix": Zabbix proxy cannot work with a Zabbix server database
zabbix-proxy exited with code 1

Update

So what I've noticed is now in the logs the proxy server complains it cannot use the zabbix database as before. So I added a new database called "zabbix_proxy" in the compose file. However, even tho the proxy server does not fail now, it still spits out errors left and right along with the zabbix server

zabbix-proxy   |    195:20240704:125809.459 cannot send proxy data to server at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-server  |    216:20240704:125810.460 cannot parse proxy data from active proxy at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-proxy   |    195:20240704:125810.461 cannot send proxy data to server at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-server  |    216:20240704:125810.916 cannot parse proxy configuration data request from active proxy at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-proxy   |    188:20240704:125810.916 cannot obtain configuration data from server at "XXX.XXX.XXX.XXX": empty string received
zabbix-server  |    214:20240704:125811.462 cannot parse proxy data from active proxy at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-proxy   |    195:20240704:125811.463 cannot send proxy data to server at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-server  |    214:20240704:125812.291 cannot send list of active checks to "XXX.XXX.XXX.XXX": host "Zabbix server" not monitored
zabbix-server  |    217:20240704:125812.464 cannot parse proxy data from active proxy at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-proxy   |    195:20240704:125812.465 cannot send proxy data to server at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-server  |    217:20240704:125813.466 cannot parse proxy data from active proxy at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-proxy   |    195:20240704:125813.467 cannot send proxy data to server at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-server  |    217:20240704:125814.367 cannot send list of active checks to "XXX.XXX.XXX.XXX": host "Zabbix server" not monitored
zabbix-server  |    216:20240704:125814.469 cannot parse proxy data from active proxy at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found
zabbix-proxy   |    195:20240704:125814.470 cannot send proxy data to server at "XXX.XXX.XXX.XXX": proxy "zabbix-proxy-mysql" not found

Is this normal?

Can I have a little help please? I am kinda feeling lost because the PHP is working, it is the proper version and NginX still refuses to see it. All this inside the container which is odd especially because I have not changed anything. I did not expect the images not working :disappointed:

dotneft commented 3 months ago
  1. The proxy "zabbix-proxy-mysql" does not exist in Zabbix configuration
  2. './zabbix_server.conf:/etc/zabbix/zabbix_server.conf' - such mount is not allowed. Use env variables.
  3. You use custom / modified Zabbix frontend php file: './CUser.php:/usr/share/zabbix/include/classes/api/services/CUser.php'
dotneft commented 3 months ago
  1. Proxy and server can not use the same DB, but you already fixed it.
bertalanimre commented 3 months ago

Thank you for your advices. Please excuse me for being confused but I don't really get the first point of yours.

2; All clear, trying it now 3; I needed to do this because we use Grafana to display the data and Grafana couldn't use the Zabbix API to authenticate and receive the source. So we used this method as a workaround. Is this resolved in any way in Version 7 ?

4; Thanks, at leas I was able to figure out one thing.

bertalanimre commented 3 months ago
  1. The proxy "zabbix-proxy-mysql" does not exist in Zabbix configuration
  2. './zabbix_server.conf:/etc/zabbix/zabbix_server.conf' - such mount is not allowed. Use env variables.
  3. You use custom / modified Zabbix frontend php file: './CUser.php:/usr/share/zabbix/include/classes/api/services/CUser.php'

Oh never mind me. I get it. I had to add the ZBX_PROXYMODE: 1 to the zabbix-proxy. Thanks. Thankfully, everything works now. :heart:

The only remaining issue was supposed to be Grafana, but it looks like it is working now out of the box.

Thank you for your help and have a great day ! :smiley: