Test Container StartAsync() results in System.OperationCanceledException

[jevonkendon]

Intermittent failure to start a Microsoft SqlServer container in an XUnit test. I setup my test case in the same way as described on the GitHub page:

public class MyTests : IAsyncLifetime
{
        public MyTests()
        {            
            var configuration = new MsSqlTestcontainerConfiguration
            {
                Password = "yourStrong(!)Password"
            };

            _container = new TestcontainersBuilder<MsSqlTestcontainer>()
                .WithDatabase(configuration)
                .Build();
    }

    public async Task InitializeAsync()
    {
        await _container.StartAsync();
    }

    public Task DisposeAsync()
    {
        return _container.DisposeAsync().AsTask();
    }

    ...
}

Results in:

System.OperationCanceledException: The operation was canceled.

System.OperationCanceledException
The operation was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
   at Docker.DotNet.ContainerOperations.ListContainersAsync(ContainersListParameters parameters, CancellationToken cancellationToken)
   at DotNet.Testcontainers.Clients.DockerContainerOperations.ByPropertyAsync(String property, String value, CancellationToken ct)
   at DotNet.Testcontainers.Clients.DockerContainerOperations.ExistsWithIdAsync(String id, CancellationToken ct)
   at DotNet.Testcontainers.Clients.TestcontainersClient.StartAsync(String id, CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.Start(String id, CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.StartAsync(CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, String ryukImage, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, String ryukImage, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, CancellationToken ct)
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(ITestcontainersConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.Create(CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.StartAsync(CancellationToken ct)
   at MyTests.InitializeAsync() in C:\MyTests\test\ATest.cs:line 129
   at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 199
   at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107

Running on Windows 10 with Docker Desktop.

Interestingly, as soon as the test exits, I see a container in Docker Desktop.

Anecdotally, if I step through the inside of StartAsync() in the debugger then it works every time. But after spending most of the day on this, I admit I'm becoming superstitious.

No, it's true! If I step through this section is works ~every time~ more often than not:

[HofmeisterAn]

Looks very unusual. It is not recommended, but can you disable the Resource Reaper for a moment? Set the environment variable TESTCONTAINERS_RYUK_DISABLED to true.

[kasparas12]

@HofmeisterAn having similar issue. While debugging tests - everything seems to be working. However when running them, container creation is hanging and Docker API returns timeout. I see testcontainers container created in Docker Desktop afterwards.

WSL2 on Windows 11 Docker Desktop

[HofmeisterAn]

Can you please share the version and your Testcontainesr for .NET configuration? @kasparas12 Do you get the same stacktrace? Did you try the suggestion above?

[kasparas12]

@HofmeisterAn my container initialization logic is this:

  _container = new TestcontainersBuilder<TestcontainersContainer>()
          .WithImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator")
          .WithName("azure-cosmos-emulator")
          .WithExposedPort(8081)
          .WithExposedPort(10251)
          .WithExposedPort(10252)
          .WithExposedPort(10253)
          .WithExposedPort(10254)
          .WithPortBinding(8081, true)
          .WithPortBinding(10251, true)
          .WithPortBinding(10252, true)
          .WithPortBinding(10253, true)
          .WithPortBinding(10254, true)
          .WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "1")
          .WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", "127.0.0.1")
          .WithEnvironment("AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE", "false")
          .WithOutputConsumer(_consumer)
          .WithWaitStrategy(Wait.ForUnixContainer()
            .UntilMessageIsLogged(_consumer.Stdout, "Started"))
           .Build();

        if (_container is not null)
        {
            await _container.StartAsync();
        }

        var mappedPort = _container?.GetMappedPublicPort(8081);
        var cosmosDbHost = $"https://{_container?.Hostname}:{mappedPort}";

        var accountEndpoint = cosmosDbHost;
        var accountKey = "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==";
        var databaseName = "ShoppingGuideDB";

        var services = new ServiceCollection();

        services.AddDbContext<ShoppingGuideDbContext>(options => options.UseCosmos(accountEndpoint, accountKey, databaseName, options =>
        {
            options.HttpClientFactory(() =>
            {
                HttpMessageHandler httpMessageHandler = new HttpClientHandler()
                {
                    ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
                };

                return new HttpClient(httpMessageHandler);
            });
            options.ConnectionMode(ConnectionMode.Gateway);
        }), ServiceLifetime.Scoped);

Stacktrace:

Stack Trace: 
AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
Int32>.GetResult(Int16 token)
SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
HttpConnection.InitialFillAsync(Boolean async)
HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
CosmosHttpClientCore.ExecuteHttpHelperAsync(HttpRequestMessage requestMessage, ResourceType resourceType, CancellationToken cancellationToken)
CosmosHttpClientCore.SendHttpHelperAsync(Func`1 createRequestMessageAsync, ResourceType resourceType, HttpTimeoutPolicy timeoutPolicy, IClientSideRequestStatistics clientSideRequestStatistics, CancellationToken cancellationToken)
GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)
TestCleanup Stack Trace
HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
CosmosHttpClientCore.ExecuteHttpHelperAsync(HttpRequestMessage requestMessage, ResourceType resourceType, CancellationToken cancellationToken)
CosmosHttpClientCore.SendHttpHelperAsync(Func`1 createRequestMessageAsync, ResourceType resourceType, HttpTimeoutPolicy timeoutPolicy, IClientSideRequestStatistics clientSideRequestStatistics, CancellationToken cancellationToken)
GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)

Testcontainers 2.1.0

[HofmeisterAn]

@kasparas12 The stacktrace does not look like it belongs to tc (and not to this issue). It looks like you are blocking the async contect with GetAwaiter().GetResult().

[jevonkendon]

Embarrassingly, after a restart of my machine it's been working flawlessly. I suspect Docker Desktop had got itself into a state. I can't reproduce it.

[HofmeisterAn]

👍 Great you solved it. I will close the issue. Do not hesitate to reopen it again if you have further issues.