MariaDB does not start

[paulz1]

I did docker pull monitoringartist/zabbix-db-mariadb today... Then MariaDB could not start now. I'm not sure it's because of pulling of new image, as there was a problem with running container even before pull.

This is an error.log from MariaDb container :

170418 10:47:03 [Note] /usr/sbin/mysqld (mysqld 10.0.30-MariaDB) starting as process 254 ... 170418 10:47:03 [Warning] Could not increase number of max_open_files to more than 4096 (request: 4407) 170418 10:47:03 [Note] InnoDB: Using mutexes to ref count buffer pool pages 170418 10:47:03 [Note] InnoDB: The InnoDB memory heap is disabled 170418 10:47:03 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 170418 10:47:03 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 170418 10:47:03 [Note] InnoDB: Compressed tables use zlib 1.2.7 170418 10:47:03 [Note] InnoDB: Using Linux native AIO 170418 10:47:03 [Note] InnoDB: Using CPU crc32 instructions 170418 10:47:03 [Note] InnoDB: Initializing buffer pool, size = 768.0M 170418 10:47:03 [Note] InnoDB: Completed initialization of buffer pool 170418 10:47:03 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode 170418 10:47:03 [ERROR] InnoDB: The system tablespace must be writable! 170418 10:47:03 [ERROR] Plugin 'InnoDB' init function returned error. 170418 10:47:03 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 170418 10:47:03 [ERROR] Failed to initialize plugins. 170418 10:47:03 [ERROR] Aborting

Does the message about "Could not increase number of max_open_files to more than 4096 (request: 4407)" is important. I thought that it could explain the problem, but it's just marked as Warning.

And, if not... what could explain the problem with "./ibdata1"? This is a part of compose file :

zabbix-db-storage:
  image: busybox:latest
  volumes:
    - /var/lib/mysql

zabbix-db:
  image: monitoringartist/zabbix-db-mariadb
  volumes:
    - /backups:/backups
  volumes_from:
    - zabbix-db-storage

[jangaraj]

It looks like a memory problem. Be sure to have at least 2GB free RAM for DB container.

[paulz1]

That was my first idea. But, the host server has 8GB de RAM.

$ free -m
             total       used       free     shared    buffers     cached
Mem:          8005        286       7719          8         10        160
-/+ buffers/cache:        115       7890
Swap:         4767          0       4767

So it's seems that there is a lot of free mem.

[jangaraj]

Could you provide docker info please?

[paulz1]

$ docker info
Containers: 17
 Running: 1
 Paused: 0
 Stopped: 16
Images: 34
Server Version: 17.04.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: xfs
 Dirs: 79
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: 
containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.818GiB
Name: domain-zabbix
ID: 6HE2:BHH2:2BIR:V42H:VN2B:H6XD:UIDW:IIIL:DWZR:AC43:SZDZ:SBRB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://domain-proxy:8080
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

[paulz1]

Just for information, it's not a problem with max_open_files. max_open_files is defined in /etc/my.cnf.d/tuning.cnf I set it to 8192. The Warning has gone, but the MariaDB still does not start with the same message.

[paulz1]

Ah, sorry, forget to precise... I change Name and Http Proxy but the both value are correct.

[paulz1]

Sorry, to ask a lot question in once, but I try to investigate on my side as well.

Could it be the rights problem? This is the list of files in /var/lib/mysql (inside of container) :

# ls -alh
total 333M
drwxr-xr-x  5 mysql mysql 4.0K Apr 18 12:03 .
drwxr-xr-x 15 root  root    38 Apr 13 21:25 ..
-rw-rw----  1   997   995  16K Apr  3 16:08 aria_log.00000001
-rw-rw----  1   997   995   52 Apr  3 16:08 aria_log_control
-rw-r--r--  1 mysql root  3.1K Apr 18 12:03 error.log
-rw-r--r--  1 mysql root  1.6K Apr 18 10:25 error.log.old
-rw-rw----  1   997   995 128M Apr 17 02:38 ib_logfile0
-rw-rw----  1   997   995 128M Apr 17 00:09 ib_logfile1
-rw-rw----  1   997   995  76M Apr 17 02:38 ibdata1
-rw-rw----  1   997   995    0 Jul 21  2016 multi-master.info
drwx------  2   997 root  4.0K Jul 21  2016 mysql
srwxrwxrwx  1   997   995    0 Apr  3 16:09 mysql.sock
drwx------  2   997   995 4.0K Jul 21  2016 performance_schema
drwx------  2   997   995  12K Sep 20  2016 zabbix

But,

# grep mysql /etc/passwd
mysql:x:998:996:MySQL server:/var/lib/mysql:/sbin/nologin

Is it normal that uid and gid of files don't correspond to uid/gid of mysql user? And is it possible that docker pull could change uid/gid?

Another question... how to fix it?

[jangaraj]

I'm not able to replicate your issue on docker 17.04 with current latest images. Could you delete all containers and run these commands please:

docker run -d -v /var/lib/mysql --name zabbix-db-storage busybox:latest
docker run \
    -d \
    --name zabbix-db \
    -v /backups:/backups \
    -v /etc/localtime:/etc/localtime:ro \
    --volumes-from zabbix-db-storage \
    --env="MARIADB_USER=zabbix" \
    --env="MARIADB_PASS=my_password" \
    monitoringartist/zabbix-db-mariadb
docker logs -f zabbix-db

[paulz1]

Hmmm... Do I understand correctly that if I delete my zabbix-db-storage container I will lost my zabbix database? At least between today and my last mysqldump (which is a little bit far from yesterday).

[jangaraj]

Do I understand correctly that if I delete my zabbix-db-storage container I will lost my zabbix database?

Yes, you will delete your zabbix DB data.

[paulz1]

Ok. So... a) at first, I connect to running container and chage the ownership of files in /var/lib/mysql chown -R mysql:mysql /var/lib/mysql it was sufficient to correct the problem. Now MariaDB starts and so zabbix container works as well (an so I could make a mysqldump of zabbix database :-) ).

b) just for invetigate This zabbix server is a VM, so clone it and the remove all containers and run the new ones with your commands. It works without problem as well. I do not print the logs output because it just works well without problem.

So the summary: a) That was a problem with the rights/ownership of /var/lib/mysql b) setting the good ownership correct the problem; c) the only strange thing is why the uid/gid of mysql user has changed in zabbix-db container (or on the storage).

I think we could close the issue. We will see if this problem will appear again in the future.

And... Thank you very-very much for your help.