[Bug]: Problems with running dotnet integration tests with testcontainers on Gitlab CI/CD

[MladenMircic]

Testcontainers version

3.8.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

amd64

.NET version

7.0.410

Docker version

-

Docker info

-

What happened?

Expected behavior Running dotnet xUnit integration tests are working correctly when run on local machine using MSSQL testcontainer. Same is expected when running them on Gitlab CI/CD using DinD service.

Actual behavior Tests are failing because Ryuk initialization has stopped.

I have seen multiple issues reported on the same topic and I have tried almost all of the suggested solutions which include setting TESTCONTAINERS_HOST_OVERRIDE: "host.docker.internal" and TESTCONTAINERS_HOST_OVERRIDE: "localhost" and disabling Ryuk altogether.

Exception when I set TESTCONTAINERS_HOST_OVERRIDE: "host.docker.internal":

Error Message:
   DotNet.Testcontainers.Containers.ResourceReaperException : Initialization has been cancelled.
  Stack Trace:
     at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, ILogger logger, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 245
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, ILogger logger, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 257
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, ILogger logger, Boolean isWindowsEngineEnabled, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 149
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 294
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 424
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 288
   at NBS.CaseRegistry.IntegrationTests.Abstractions.IntegrationTestWebAppFactory.InitializeAsync() in /builds/km-nbs-projects/case-registry/tests/NBS.CaseRegistry.IntegrationTests/Abstractions/IntegrationTestWebAppFactory.cs:line 57
  Failed NBS.CaseRegistry.IntegrationTests.Mails.CreateMailFromActTests.CreateMailFromAct_ShouldBeValid_WhenUsingGlobalCategory [1 ms]

Exception when I set TESTCONTAINERS_HOST_OVERRIDE: "localhost":

[testcontainers.org 00:00:00.09] Connected to Docker:
  Host: tcp://docker:2375/
  Server Version: 26.1.4
  Kernel Version: 5.15.154+
  API Version: 1.45
  Operating System: Alpine Linux v3.20 (containerized)
  Total Memory: 7.77 GB
[testcontainers.org 00:00:00.12] Docker config "/root/.docker/config.json" not found
[testcontainers.org 00:00:00.12] Searching Docker registry credential in Auths
[testcontainers.org 00:00:00.12] Docker registry credential https://index.docker.io/v1/ not found
[testcontainers.org 00:00:01.38] Docker image testcontainers/ryuk:0.6.0 created
[testcontainers.org 00:00:01.57] Docker container 5af8e5e6281c created
[testcontainers.org 00:00:01.62] Start Docker container 5af8e5e6281c
[testcontainers.org 00:00:01.93] Wait for Docker container 5af8e5e6281c to complete readiness checks
[testcontainers.org 00:00:01.93] Docker container 5af8e5e6281c ready
[testcontainers.org 00:01:02.10] Stop Docker container 5af8e5e6281c

And then every test throws the same exception:

Docker.DotNet.DockerContainerNotFoundException : Docker API responded with status code=NotFound, response={"message":"No such container: 5af8e5e6281ccd84a10b1add6572269cf066feac2d24b744f1f3143a70c7946a"}

Exception when I disable Ryuk:

Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)

Relevant log output

No response

Additional information

Container initialization factory

public sealed class IntegrationTestWebAppFactory : WebApplicationFactory<Program>, IAsyncLifetime
{
    private readonly MsSqlContainer _dbContainer =
        new MsSqlBuilder()
            .Build();

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
        {
            services.RemoveAll(typeof(DbContextOptions<MyDbContext>));

            var sqlConnectionStringBuilder =
                new SqlConnectionStringBuilder(_dbContainer.GetConnectionString())
                {
                    TrustServerCertificate = true,
                    Encrypt = false,
                    MultiSubnetFailover = true
                };

            services.AddDbContext<MyDbContext>(options =>
            {
                options.UseSqlServer(
                    sqlConnectionStringBuilder.ConnectionString,
                    sqlServerOptions => sqlServerOptions.CommandTimeout(20));
            });
        });
    }

    public async Task InitializeAsync()
    {
        await _dbContainer.StartAsync();
    }

    public async new Task DisposeAsync()
    {
        await _dbContainer.StopAsync();
    }
}

I have also tried binding to port 1433 for MSSQL

private readonly MsSqlContainer _dbContainer =
        new MsSqlBuilder()
            .WithPortBinding(1433, true)
            .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(1433))
            .Build();

I am absolutely sure that I have misconfigured something, but I can't quite figure it out. Any help would be highly appreciated!

[HofmeisterAn]

Phew, it has been a while since I configured GitLab CI/CD with Testcontainers. IIRC it is not necessary to set the TESTCONTAINERS_HOST_OVERRIDE environment variable. Can you please remove all the environment variable configurations and follow this and/or this approach (without the Java-specific steps) to configure the DinD service? If you still run into issues, please follow up with the error.

[MladenMircic]

It was a problem connecting Hangfire library to the test environment. It is now fixed, thank you :)

[HofmeisterAn]

Ok, this looks much better. In the logs, you can see that Testcontainers can pull, run, and access Ryuk, and at least pull and run MSSQL.

According to the logs, it appears that you have overwritten the default wait strategy. Please do not. Your chosen wait strategy may indicate readiness too early (the network port is available before the actual service in the container is running). Please use the default builder configuration.

IIRC it is necessary to remove more DI registered services. Please refer to this guide for further details.

These days, I prefer to provide the connection string via the application settings and not mess around with the DI registered services (see Set Redis connection string).