search

MariaDB / mariadb-docker

Docker Official Image packaging for MariaDB
https://mariadb.org
GNU General Public License v2.0
759 stars 438 forks source link

mysqld: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit #454

Closed zierbeek closed 1 year ago

zierbeek commented 1 year ago

Hi all,

I have replaced image 10.5 with 10.8 on ubuntu 20.04 LTS running on a VM. This is to support photoprism. The mariadb was running happily before the upgrade but know, it refuses to start. I think I should have followed a guide of mariadb instead of just changing the tag. I can drop the database, if that would fix it.

Docker-Compose

    ## If MariaDB gets stuck in a restart loop, this points to a memory or filesystem issue:
    ## https://docs.photoprism.app/getting-started/troubleshooting/#fatal-server-errors
    restart: unless-stopped
    container_name: mariadb
    image: mariadb:10.8
    security_opt: # see https://github.com/MariaDB/mariadb-docker/issues/434#issuecomment-1136151239
      - seccomp:unconfined
      - apparmor:unconfined
    command: mysqld --innodb-buffer-pool-size=512M --transaction-isolation=READ-COMMITTED --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --max-connections=512 --innodb-rollback-on-timeout=OFF --innodb-lock-wait-timeout=120
    ## Never store database files on an unreliable device such as a USB flash drive, an SD card, or a shared network folder:
    volumes:
      - "./database:/var/lib/mysql" # DO NOT REMOVE
    environment:
      MARIADB_AUTO_UPGRADE: "1"
      MARIADB_INITDB_SKIP_TZINFO: "1"
      MARIADB_DATABASE: "photoprism"
      MARIADB_USER: "***"
      MARIADB_PASSWORD: "***"
      MARIADB_ROOT_PASSWORD: "***"

LOGS

2022-08-05 07:39:16+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-08-05 07:39:16+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-08-05 07:39:17+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2022-08-05 07:39:17+00:00 [Note] [Entrypoint]: Starting temporary server
2022-08-05 07:39:17+00:00 [Note] [Entrypoint]: Waiting for server startup
2022-08-05  7:39:17 0 [Note] mysqld (server 10.8.3-MariaDB-1:10.8.3+maria~jammy) starting as process 48 ...
2022-08-05  7:39:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-08-05  7:39:18 0 [Note] InnoDB: Number of transaction pools: 1
2022-08-05  7:39:18 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-08-05  7:39:18 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-08-05  7:39:18 0 [Warning] mysqld: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2022-08-05  7:39:18 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-08-05  7:39:18 0 [Note] InnoDB: Initializing buffer pool, total size = 512.000MiB, chunk size = 8.000MiB
2022-08-05  7:39:18 0 [Note] InnoDB: Completed initialization of buffer pool
2022-08-05  7:39:18 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.9.
2022-08-05  7:39:18 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2022-08-05  7:39:18 0 [Note] InnoDB: Starting shutdown...
2022-08-05  7:39:18 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-08-05  7:39:18 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-08-05  7:39:18 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-08-05  7:39:18 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-08-05  7:39:18 0 [ERROR] Aborting
dr-m commented 1 year ago

Does the advice given in https://mariadb.com/kb/en/systemd/#configuring-limitmemlock work for you? Starting with MariaDB/server@783625d78f5072a4f986f079628535dc70d7e99f (MariaDB Server 10.6.0) uses the io_uring a.k.a. liburing interface for asynchronous I/O when it is enabled at compilation time. Older versions only support libaio, which has a different set of limitations.

However, the main problem appears to be that you are trying to upgrade a database without a prior clean shutdown of the old version:

2022-08-05  7:39:18 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.9.

The InnoDB log format was rewritten in MariaDB/server@685d958e38b825ad9829be311f26729cccf37c46 and just like with MariaDB Server 10.2 or MySQL 5.7 or 8.0, an upgrade from an earlier version is only possible after a proper shutdown of the earlier server.

Can you post the logs from the 10.5 server? Was it shut down correctly?

zierbeek commented 1 year ago

Hi there! I did indeed probably not shut it down correctly due to lack of knowledge. I have reverted to 10.5 and done accordingly, set it back to 10.8 and it did upgrade without any issue.

Should is still set that limitmemlock? To what value should. set it? should the file look like: 

[Service]

LimitMEMLOCK=2M

Also not able to run a systemctl command inside a docker container. 

2022-08-05 09:07:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-08-05 09:07:26+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-08-05 09:07:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-08-05 09:07:27+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2022-08-05  9:07:27 0 [Note] mysqld (server 10.8.3-MariaDB-1:10.8.3+maria~jammy) starting as process 1 ...
2022-08-05  9:07:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-08-05  9:07:27 0 [Note] InnoDB: Number of transaction pools: 1
2022-08-05  9:07:27 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-08-05  9:07:27 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-08-05  9:07:27 0 [Warning] mysqld: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2022-08-05  9:07:27 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-08-05  9:07:27 0 [Note] InnoDB: Initializing buffer pool, total size = 512.000MiB, chunk size = 8.000MiB
2022-08-05  9:07:27 0 [Note] InnoDB: Completed initialization of buffer pool
2022-08-05  9:07:27 0 [Note] InnoDB: 128 rollback segments are active.
2022-08-05  9:07:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-08-05  9:07:27 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-08-05  9:07:27 0 [Note] InnoDB: log sequence number 18087938336; transaction id 22872318
2022-08-05  9:07:27 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-08-05  9:07:27 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-08-05  9:07:27 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-08-05  9:07:27 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-08-05  9:07:27 0 [Note] Server socket created on IP: '::'.
2022-08-05  9:07:27 0 [Warning] 'proxies_priv' entry '@% root@c242040e456d' ignored in --skip-name-resolve mode.
2022-08-05  9:07:27 0 [Note] mysqld: ready for connections.
Version: '10.8.3-MariaDB-1:10.8.3+maria~jammy'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
grooverdan commented 1 year ago

Hi there! I did indeed probably not shut it down correctly due to lack of knowledge.

I'm curious how you did shut it down (both the right and wrong way) so maybe I can document it better. Maybe our server message could say something more specific about the revert, clean shutdown and then try again.

I have reverted to 10.5 and done accordingly, set it back to 10.8 and it did upgrade without any issue.

I'm glad that went smoothly.

Should is still set that limitmemlock?

While you are on an older Ubuntu 20.04 kernel its still required.

To what value should. set it?

from the error message (262144 bytes required)

should the file look like:

The systemd recommendation is generic from the server. What you'll need is a docker compose ulimit of memlock

Technically the liburing support start in 10.8 in the container images as that's when it switched to Ubuntu-22.04 as the base image which is the first version that had the user space support. < 10.8 are all 20.04 focal based an use libaio.

zierbeek commented 1 year ago

well, the wrong way was just changing the docker-compose tag from mariadb_10.5 to 10.8 and docker-compose pull and up. The right way was docker-compose down, change tag, docker-compose pull.

i have set it like this, but doesn't seem to solve that warning

  mariadb:
    ## If MariaDB gets stuck in a restart loop, this points to a memory or filesystem issue:
    ## https://docs.photoprism.app/getting-started/troubleshooting/#fatal-server-errors
    restart: unless-stopped
    container_name: mariadb
    image: mariadb:10.8
    security_opt: # see https://github.com/MariaDB/mariadb-docker/issues/434#issuecomment-1136151239
      - seccomp:unconfined
      - apparmor:unconfined
    command: mysqld --innodb-buffer-pool-size=512M --transaction-isolation=READ-COMMITTED --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --max-connections=512 --innodb-rollback-on-timeout=>
    ## Never store database files on an unreliable device such as a USB flash drive, an SD card, or a shared network folder:
    ulimits:
      nproc: 262144
    volumes:
      - "./database:/var/lib/mysql" # DO NOT REMOVE

this config isn't working as well. No error but no change in warning

    ulimits:
      nofile:
        soft: "262144"
        hard: "262144"
grooverdan commented 1 year ago

its a memlock limit, not a nofile or nproc. ( I also don't know if the limits are kbytes or bytes in the compose format).

zierbeek commented 1 year ago

its a memlock limit, not a nofile or nproc. ( I also don't know if the limits are kbytes or bytes in the compose format).

can confirm that this removes the error. Could you please explain what this does and why this was needed? This was in conjunction with photoprism

ulimits:
      memlock: "262144"
grooverdan commented 1 year ago

This was only a warning, so it wasn't fatal. The raised limit allows a process to lock the specified size of memory for its exclusive use (so the kernel cannot swap this memory out). It was required to use uring with Linux kernels up until 5.12 after which the restriction was relaxed.