[Bug]: MySQL 8.0.28 Container Fails to Start with Testcontainers 3.8.0

[macalbert]

Testcontainers version

3.8.0

Using the latest Testcontainers version?

Yes

Host OS

Windows/Ubuntu

Host arch

x86/ARM

.NET version

8.0

Docker version

Docker Desktop 4.28.0 (139021)

Docker info

❯ docker info
Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.48GiB
 Name: docker-desktop
 ID: 231921ac-6bb9-409e-b0d6-ae72b231b261
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

What happened?

When attempting to start a MySQL 8.0.28 container using Testcontainers 3.8.0, the operation fails with a Docker.DotNet.DockerApiException, indicating that the container is not running. This issue occurs during the execution of integration tests, specifically when trying to establish a connection to the MySQL container.

I'm including a simplified test scenario that demonstrates the problem. The test attempts to start a MySQL 8.0.28 container and then perform two operations: opening a database connection and executing a simple SQL script. Both tests highlight the failure to start the container, as indicated by the Docker.DotNet.DockerApiException.

namespace Testcontainers.MySql;

using MySqlConnector;
using System.Data;
using System.Data.Common;

public class MySqlContainerTest : IAsyncLifetime
{
    private readonly MySqlContainer _mySqlContainer;

    public MySqlContainerTest()
    {
        _mySqlContainer = new MySqlBuilder().WithImage("mysql:8.0.28")
                                            .Build();
    }

    public Task InitializeAsync()
    {
        return _mySqlContainer.StartAsync();
    }

    public Task DisposeAsync()
    {
        return _mySqlContainer.DisposeAsync().AsTask();
    }

    [Fact]
    public void Should_Connect_When_CreateConnectionToContainer()
    {
        // Arrage
        using DbConnection connection = new MySqlConnection(_mySqlContainer.GetConnectionString());

        // Act
        connection.Open();

        // Assert
        Assert.Equal(ConnectionState.Open, connection.State);
    }

    [Fact]
    public async Task Should_RunSuccessful_When_ExecScript()
    {
        // Arrage
        const string scriptContent = "SELECT 1;";

        // Act
        var execResult = await _mySqlContainer.ExecScriptAsync(scriptContent)
                                              .ConfigureAwait(true);

        // Assert
        Assert.True(0L.Equals(execResult.ExitCode), execResult.Stderr);
        Assert.Empty(execResult.Stderr);
    }
}

Relevant log output

2024-03-20 11:58:05 2024-03-20T10:58:05.673661Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.28) initializing of server in progress as process 42
2024-03-20 11:58:05 2024-03-20T10:58:05.684913Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-03-20 11:58:06 2024-03-20T10:58:06.012863Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-03-20 11:58:07 2024-03-20T10:58:07.036748Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2024-03-20 11:58:05 2024-03-20 10:58:05+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.28-1debian10 started.
2024-03-20 11:58:05 2024-03-20 10:58:05+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-03-20 11:58:05 2024-03-20 10:58:05+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.28-1debian10 started.
2024-03-20 11:58:05 2024-03-20 10:58:05+00:00 [Note] [Entrypoint]: Initializing database files
2024-03-20 11:58:09 2024-03-20 10:58:09+00:00 [Note] [Entrypoint]: Database files initialized
2024-03-20 11:58:09 2024-03-20 10:58:09+00:00 [Note] [Entrypoint]: Starting temporary server
2024-03-20 11:58:09 mysqld: Error on realpath() on '/var/lib/mysql-files' (Error 2 - No such file or directory)
2024-03-20 11:58:09 2024-03-20T10:58:09.680732Z 0 [ERROR] [MY-010095] [Server] Failed to access directory for --secure-file-priv. Please make sure that directory exists and is accessible by MySQL Server. Supplied value : /var/lib/mysql-files
2024-03-20 11:58:09 2024-03-20 10:58:09+00:00 [ERROR] [Entrypoint]: Unable to start server.

Additional information

No response

[HofmeisterAn]

Does it fail with 3.7.0 too? Our CI should use the latest patch version too 🤔. Maybe it is a regression from uploading the /etc/mysql/my.cnf file (#1116) /cc @0xced. I am wondering about:

mysqld: Error on realpath() on '/var/lib/mysql-files' (Error 2 - No such file or directory)

I can take a look at it end of next week.

[macalbert]

I've tested it with 3.7.0, and it's working well. It seems like the issue might not be related to the version itself. Maybe the regression from uploading the /etc/mysql/my.cnf file, as mentioned in #1116, doesn't affect this version?

I'll keep an eye out for any related errors, but so far, so good.

[0xced]

I just submitted #1144 which should address this issue.

Note that this problem only exist for mysql:8.0.28 and earlier versions. mysql:8.0.29 onwards images have an empty /var/lib/mysql-files directory and the containers start fine.

This is also the reason why this was not caught by continuous integration since the default image against which the test are run is mysql:8.0 which currently corresponds to mysql:8.0.36.

[HofmeisterAn]

@0xced Thanks for the PR. @macalbert you can work around the issue until a new version is available by adding the following line to the builder configuration:

WithResourceMapping(Array.Empty<byte>(), "/var/lib/mysql-files/gh-issue-1142")

This will create the directory upon container start.