latest image has no connection

[hanna-eismant]

I have an CI/CD that is configured to run integration tests and it uses mysql docker image for it. Previously it run without problem, but today all tests are failed cause cannot connect to database.

For database startup I have follow configuration. As you can see it is not specified with image tag so it uses latest.

version: '3.9'

services:
  db:
    image: mysql
    networks:
      - doky-test-net
    ports:
      - "3306:3306"
    restart: unless-stopped
    command: '--default-authentication-plugin=mysql_native_password'
    cap_add:
      - SYS_NICE
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: doky-test
      MYSQL_USER: doky-test
      MYSQL_PASSWORD: doky-test

networks:
  doky-test-net:

After investigation I found out that success builds uses mysql:8.3.0 (sha256:9de9d54fecee6253130e65154b930978b1fcc336bcc86dfd06e89b72a2588ebe) and failed uses mysql:8.4.0 (sha256:f7a8e140a7d6d1e6e0c99eeb0489c50a186ee4ac44ff55323a176529b9a43d33). I changes my test-db.yml to use image mysql:8.3.0 and it works.

The service is started without error. There are logs from CI/CD:

Step 1/3: Start Database (Docker Compose)
16:38:23   Starting docker-compose for teamcity/test-db.yml
16:38:23   Starting: /bin/sh -c docker-compose  -f "teamcity/test-db.yml" up -d
16:38:23   in directory: /mnt/agent/work/bc4e634d0c6d276e
16:38:24   Creating network "teamcity_doky-test-net" with the default driver
16:38:24   Pulling db (mysql:)...
16:38:24   latest: Pulling from library/mysql
16:38:37   Digest: sha256:f7a8e140a7d6d1e6e0c99eeb0489c50a186ee4ac44ff55323a176529b9a43d33
16:38:37   Status: Downloaded newer image for mysql:latest
16:38:37   Creating teamcity_db_1 ...
16:38:41   Creating teamcity_db_1 ... done
16:38:41   Process exited with code 0
16:38:42   docker-compose has started successfully
16:38:43   Registering environment variable TEAMCITY_DOCKER_NETWORK=teamcity_doky-test-net

And corresponding messages in tests:

Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. 
The driver has not received any packets from the server.
Caused by: java.net.ConnectException: Connection refused

[amsterreynolds-123]

I'm having the same issue following upgrade to mysql:8.4.0, however rolling back to mysql:8.3.0 doesn't work (only permitted between patch releases.)

Container logs:

2024-05-01T15:18:35.265911Z 0 [System] [MY-015015] [Server] MySQL Server - start. 2024-05-01T15:18:35.665802Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1 2024-05-01T15:18:35.681111Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-05-01T15:18:36.340873Z 1 [ERROR] [MY-014061] [InnoDB] Invalid MySQL server downgrade: Cannot downgrade from 80400 to 80300. Downgrade is only permitted between patch releases. mysqld: Can't open file: 'mysql.ibd' (errno: 0 - ) 2024-05-01T15:18:36.655654Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2024-05-01T15:18:36.656539Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. 2024-05-01T15:18:36.656732Z 0 [ERROR] [MY-010119] [Server] Aborting 2024-05-01T15:18:36.659590Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.3.0) MySQL Community Server - GPL. 2024-05-01T15:18:36.659632Z 0 [System] [MY-015016] [Server] MySQL Server - end. 2024-05-01 15:18:39+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started. 2024-05-01 15:18:39+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' 2024-05-01 15:18:39+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started. '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock' 2024-05-01T15:18:40.027563Z 0 [System] [MY-015015] [Server] MySQL Server - start. 2024-05-01T15:18:40.412605Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1 2024-05-01T15:18:40.426081Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-05-01T15:18:40.979824Z 1 [ERROR] [MY-014061] [InnoDB] Invalid MySQL server downgrade: Cannot downgrade from 80400 to 80300. Downgrade is only permitted between patch releases. mysqld: Can't open file: 'mysql.ibd' (errno: 0 - ) 2024-05-01T15:18:41.308533Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2024-05-01T15:18:41.308969Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. 2024-05-01T15:18:41.309086Z 0 [ERROR] [MY-010119] [Server] Aborting 2024-05-01T15:18:41.311853Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.3.0) MySQL Community Server - GPL. 2024-05-01T15:18:41.311941Z 0 [System] [MY-015016] [Server] MySQL Server - end.

Any assistance will be appreciated as I'm at a dead end.

[yonran]

mysql:8.4.0 gives this message with --default-authentication-plugin=mysql_native_password before exiting: [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.:

docker run --name=my-mysql --rm -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test  mysql:8.4.0 --default-authentication-plugin=mysql_native_password
…
2024-05-01T16:48:44.890653Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-05-01T16:48:44.891100Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-05-01T16:48:44.891137Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T16:48:46.079987Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.

[zarevskaya]

mysql:8.4.0 gives this message with --default-authentication-plugin=mysql_native_password before exiting: [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.:

docker run --name=my-mysql --rm -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test  mysql:8.4.0 --default-authentication-plugin=mysql_native_password
…
2024-05-01T16:48:44.890653Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-05-01T16:48:44.891100Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-05-01T16:48:44.891137Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T16:48:46.079987Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.

Yes, the same here. A real disaster... You are not alone. ;)

[LaurentGoderre]

Sort of related to #1039

[amsterreynolds-123]

mysql:8.4.0 gives this message with --default-authentication-plugin=mysql_native_password before exiting: [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.:

docker run --name=my-mysql --rm -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test  mysql:8.4.0 --default-authentication-plugin=mysql_native_password
…
2024-05-01T16:48:44.890653Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-05-01T16:48:44.891100Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-05-01T16:48:44.891137Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T16:48:46.079987Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.

Awesome - thanks!

[yosifkit]

Deprecation and Removal Notes Important Change: The deprecated mysql_native_password authentication plugin is now disabled by default. It can be enabled by starting MySQL with the new --mysql-native-password=ON server option, or by adding mysql_native_password=ON to the [mysqld] section of your MySQL configuration file.

For more information, see Native Pluggable Authentication. (Bug #36337893)

- https://dev.mysql.com/doc/relnotes/mysql/8.4/en/news-8-4-0.html

[yosifkit]

I would recommend reading at least the full Deprecation and Removal Notes section of those release notes in case you are using now deprecated or removed features, but all of it might contain important information relevant to your particular deployment.

https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions/ would also be good to read about when updates or breaking changes happen.

---

TL;DR: add --mysql-native-password=ON to your args ~along with~ and remove the --default-authentication-plugin=mysql_native_password that you already have or use mysql:8.3 (the image won't be updated anymore). If your database files already upgraded, then you'll have to dump from 8.4 and load on your target.

---

I'd also recommend moving away from mysql:latest or mysql (implied :latest) and instead picking an actively supported version that suits your needs like mysql:8.0 or mysql:8.4 (as the two versions currently supported).

[hanna-eismant]

@yosifkit Thanks for clarification. My mistake is that I used latest and was not notified that version was upgraded without additional preparations.

[zarevskaya]

@yosifkit Thanks for clarification. My mistake is that I used latest and was not notified that version was upgraded without additional preparations.

Yep, the same here with Ghost (CMS), what nightmare. I'm guilty.

[thebigbone]

I would recommend reading at least the full Deprecation and Removal Notes section of those release notes in case you are using now deprecated or removed features, but all of it might contain important information relevant to your particular deployment.

https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions/ would also be good to read about when updates or breaking changes happen.

TL;DR: add --mysql-native-_password=ON to your args along with the --default-authentication-plugin=mysql_native_password that you already have or use mysql:8.3 (the image won't be updated anymore). If your database files already upgraded, then you'll have to dump from 8.4 and load on your target.

I'd also recommend moving away from mysql:latest or mysql (implied :latest) and instead picking an actively supported version that suits your needs like mysql:8.0 or mysql:8.4 (as the two versions currently supported).

A little correction here. I was facing the same issue on the latest image and just by adding --mysql-native-password=ON to the command (you have a typo there) I was able to start mysql. If you add the deprecated command along with it, it will spit out the same error and refuse to start.

TLDR: Remove the --default-authentication-plugin=mysql_native_password and add --mysql-native-password=ON command.

[tianon]

I believe the upstream intent is to completely remove the functionality, so don't be surprised when you find that can again after kicking it down the road (again). :sweat_smile:

[yonran]

I'd also recommend moving away from mysql:latest or mysql (implied :latest) and instead picking an actively supported version that suits your needs like mysql:8.0 or mysql:8.4 (as the two versions currently supported).

fyi the the documentation at https://hub.docker.com/_/mysql currently has an example showing mysql:latest with --default-authentication-plugin=mysql_native_password:

# Use root/example as user/password credentials
version: '3.1'

services:

  db:
    image: mysql
    # NOTE: use of "mysql_native_password" is not recommended: https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-caching-sha2-password
    # (this is just an example, not intended to be a production configuration)
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

[tianon]

Good callout, thank you! https://github.com/docker-library/docs/pull/2443