zombie [mysqladmin] processes in branch rel-10_1

Monviech commented 2 years ago

Issue description

ISSUE: mysqladmin zombie processes are appearing minutely that aren't reaped. The parent process is mysqld.

The operating system is Ubuntu Server LTS 20.04.3 LTS
docker-compose version 1.29.2, build 5becea4c docker-py version: 5.0.0 CPython version: 3.7.10 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
Server: Docker Engine - Community Engine: Version: 20.10.12 API version: 1.41 (minimum version 1.12) Go version: go1.16.12 Git commit: 459d0df Built: Mon Dec 13 11:43:42 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.19.0 GitCommit: de40ad0
Git release rel_10-1 Otobo Docker runs with Otobo Version 10.0.15

Steps to reproduce the issue

Install VM with Ubuntu Server 20.04.3 LTS
cd /opt
git clone https://github.com/RotherOSS/otobo-docker.git --branch rel-10.1 --single-branch
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
cp /opt/otobo-docker/.docker_compose_env_http /opt/otobo-docker/.env
vi .env -> Set Database password
docker-compose up -d
Follow the Otobo Installer.pl until the end.
Log into root@localhost
zombie processes are appearing from now on
docker-compose restart -> all zombie processes get sigterm, but they reappear shortly

What's the expected result?

no zombie processes like in rel-10.0

What's the actual result?

minutely theres an additional zombie process in rel-10.1. The parent process is mysqld, the orphaned child is mysqladmin.

Additional details / screenshots

otobo_zombie_01 t

Monviech commented 2 years ago

Temporary Fix:

Commenting out line 38 healthcheck: 39 test: mysqladmin -p${OTOBO_DB_ROOT_PASSWORD:?err} ping -h localhost in otobo-base.yml solves the problem with zombie processes. The healthcheck of the mariadb 10.5 container doesn't work properly somehow.

bschmalhofer commented 2 years ago

Good catch. I started up the OTOBO containers on my Ubuntu devel machine and looked for those zombie processes with ps aux | grep Z . But I did not see any. Then I tried running the health check command within the running db container.

mysql@9f936ffd28b9:/$ mysqladmin -h localhost ping
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'mysql'@'localhost' (using password: NO)'
mysql@9f936ffd28b9:/$ echo $?
0
mysql@9f936ffd28b9:/$ mysqladmin -h web ping
mysqladmin: connect to server at 'web' failed
error: 'Can't connect to MySQL server on 'web' (115)'
Check that mysqld is running and that the socket: '/run/mysqld/mysqld.sock' exists!
mysql@9f936ffd28b9:/$ echo $?
1
mysql@9f936ffd28b9:/$ mysqladmin -h db ping
mysqladmin: connect to server at 'db' failed
error: 'Access denied for user 'mysql'@'172.18.0.4' (using password: NO)'
mysql@9f936ffd28b9:/$ echo $?
0
mysql@9f936ffd28b9:/$

This looks sensible. The exit code is 0 when the command is called with a host where MariaDB is running and 1 where not. I left out the password as this does not make sense when the user is not passed.

The zombie processes can't really be the fault of the command anyways. The processes do exit as otherwise they wouldn't be zombies. Therefore it must be Docker or Docker Compose itself, who are not properly waiting on their child processed.

Another idea is that this could be some kind of timeout issue. When I call mysqladmin -h gibtsnicht ping then the process seems to hang. Maybe docker is giving up on reaping the child processes and when they eventually die they become zombies. Could you try:

healthcheck: test: mysqladmin -h db ping

in your otobo-base.yml file ?

Monviech commented 2 years ago

Changing the healthcheck to

healthcheck: test: mysqladmin -h db ping

in the otobo-base.yml file doesnt create zombies anymore.

The health of the db container is now healthy:

1134c474098d mariadb:10.5 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp otobo_db_1

bschmalhofer commented 2 years ago

Cool, so it looks like this is DNS-related where in some installations localhost is known and in other instances in unknown. I will adapt the health check. This will be released in OTOBO 10.1 and in the next patch level release of OTOBO 10.0 if there is one.

bschmalhofer commented 2 years ago

Merged the PR. A quick test showed health status of the service 'db'. No zombies were seen. Clsoing this issue.

RotherOSS / otobo-docker