search

testcontainers / testcontainers-dotnet

A library to support tests with throwaway instances of Docker containers for all compatible .NET Standard versions.
https://dotnet.testcontainers.org
MIT License
3.73k stars 262 forks source link

[Bug]: Test Container locks up testhost after Windows Update August update #998

Closed minoseah629 closed 11 months ago

minoseah629 commented 12 months ago

Testcontainers version

3.3.0 and 3.5.0

Using the latest Testcontainers version?

Yes

Host OS

Windows 11

Host arch

x64

.NET version

7.0.400

Docker version

Client:
 Cloud integration: v1.0.35-desktop+001
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:36:24 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.22.1 (118664)
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 39
  Running: 2
  Paused: 0
  Stopped: 37
 Images: 54
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 15.47GiB
 Name: docker-desktop
 ID: 1a6d71d1-9038-4cf4-b6ae-7a1a745742f9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?

run a test in visual studio or dotnet test 1-2 tests would pass fine test runner tries to continue running test, but is currently locked up. test running is locked up due to WerFault.exe is catching an error and reporting problem to Windows

image

also happening on coworker's machine.

can provide video if requested

Relevant log output

from dotnet logs, 

logs did say testcontainer stopped before 2 or 3rd test starts.

Additional information

No response

HofmeisterAn commented 12 months ago

I have not noticed any issues on my Windows test environment.

test runner tries to continue running test, but is currently locked up. test running is locked up due to WerFault.exe is catching an error and reporting problem to Windows

Does this imply that the test keeps running infinite, or is there something causing the test process to fail (can you share the WerFault.exe report)? Could you please look into the output of the docker ps and docker inspect command? My initial assumption is that the dependent container may not be initializing correctly, causing Testcontainers to wait for the readiness check confirmation (can you share the container builder configuration).

Occasionally, resetting Docker Desktop to its default factory settings can also be helpful 😅.

minoseah629 commented 11 months ago

looks like testhost is the program that is using werfault.

image

at the start of the all of the test run, the test container starts up fine.

image

but when the 2nd or 3rd test of the whole test run, the container i am starting up with testcontainer is no longer shown in docker desktop.

i also put ConsoleLogger.Instance.DebugLogLevelEnabled = true; but not seeing logging info

HofmeisterAn commented 11 months ago

I am sorry, I do not think I can help much with this information. Have you tried to reset Docker Desktop? Are you possibly exceeding the resources allocated to your test host?

but not seeing logging info

Run the tests with Debug configuration and check the test / debug window in Visual Studio (View > Output).

In addition to that, try to run dotnet test --verbosity detailed to get a verbose output.

minoseah629 commented 11 months ago

i have not reset docker desktop. i want to preserve my existing containers. trying code modifications

will provide a dotnet test -v:diag shortly

minoseah629 commented 11 months ago

Microsoft (R) Test Execution Command Line Tool Version 17.7.1 (x64) Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. [testcontainers.org 00:00:00.05] Connected to Docker: Host: npipe://./pipe/docker_engine Server Version: 24.0.5 Kernel Version: 5.15.90.1-microsoft-standard-WSL2 API Version: 1.43 Operating System: Docker Desktop Total Memory: 15.47 GB InitializeAsync Called [testcontainers.org 00:00:00.17] Docker container 8e84a8c7fd52 created [testcontainers.org 00:00:00.23] Start Docker container 8e84a8c7fd52 [testcontainers.org 00:00:01.53] Wait for Docker container 8e84a8c7fd52 to complete readiness checks [testcontainers.org 00:00:01.54] Docker container 8e84a8c7fd52 ready [testcontainers.org 00:00:01.59] Docker container d5e4bb66c8f0 created [testcontainers.org 00:00:01.60] Start Docker container d5e4bb66c8f0 [testcontainers.org 00:00:01.92] Wait for Docker container d5e4bb66c8f0 to complete readiness checks [testcontainers.org 00:00:01.93] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container d5e4bb66c8f0 [testcontainers.org 00:00:03.05] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container d5e4bb66c8f0 [testcontainers.org 00:00:04.19] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container d5e4bb66c8f0 [testcontainers.org 00:00:04.32] Docker container d5e4bb66c8f0 ready Fi=000,Fa=000,Pe=000 # > FEATURE: GetRecentlyAccessedForUserHandlerTests Before Start Fi=000,Fa=000,Pe=001 # 1> SCENARIO: Validates QueryResults RecentlyAccessedWorkflow ReturnsCreatedWorkflow Fi=000,Fa=000,Pe=001 # 1> STEP 1/6: GIVEN There Is A Person... Fi=000,Fa=000,Pe=001 # 1> STEP 1/6: GIVEN There Is A Person (Passed after 867ms) Fi=000,Fa=000,Pe=001 # 1> STEP 2/6: AND A Workflow Exists... Fi=000,Fa=000,Pe=001 # 1> STEP 2/6: AND A Workflow Exists (Passed after 409ms) Fi=000,Fa=000,Pe=001 # 1> STEP 3/6: AND The Workflow Has A Version... Fi=000,Fa=000,Pe=001 # 1> STEP 3/6: AND The Workflow Has A Version (Passed after 1s 158ms) Fi=000,Fa=000,Pe=001 # 1> STEP 4/6: AND A Recently Accessed Exists... Fi=000,Fa=000,Pe=001 # 1> STEP 4/6: AND A Recently Accessed Exists (Passed after 92ms) Fi=000,Fa=000,Pe=001 # 1> STEP 5/6: WHEN Query Is Invoked... Before End Fi=000,Fa=000,Pe=001 # 1> STEP 5/6: WHEN Query Is Invoked (Passed after 572ms) Fi=000,Fa=000,Pe=001 # 1> STEP 6/6: THEN QueryResult IncludesRecentlyCreatedWorkflow... Fi=000,Fa=000,Pe=001 # 1> STEP 6/6: THEN QueryResult IncludesRecentlyCreatedWorkflow (Passed after 13ms) Fi=001,Fa=000,Pe=000 # 1> SCENARIO RESULT: Passed after 3s 211ms After Start [testcontainers.org 00:00:24.40] Delete Docker container d5e4bb66c8f0 After End InitializeAsync Called [testcontainers.org 00:00:26.37] Docker container 2c41df1aab76 created [testcontainers.org 00:00:26.43] Start Docker container 2c41df1aab76 [testcontainers.org 00:00:27.26] Wait for Docker container 2c41df1aab76 to complete readiness checks [testcontainers.org 00:00:27.29] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 2c41df1aab76 [testcontainers.org 00:00:28.55] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 2c41df1aab76 [testcontainers.org 00:00:29.82] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 2c41df1aab76 [testcontainers.org 00:00:31.14] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 2c41df1aab76 [testcontainers.org 00:00:32.36] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 2c41df1aab76 [testcontainers.org 00:00:33.65] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 2c41df1aab76 [testcontainers.org 00:00:33.86] Docker container 2c41df1aab76 ready Fi=001,Fa=000,Pe=000 # > FEATURE: GetApplicationByIdQueryTests Before Start Fi=001,Fa=000,Pe=001 # 2> SCENARIO: Validates QueryResults WhenDataIsSetup AndShowArchived [showArchived: "False"] Fi=001,Fa=000,Pe=001 # 2> STEP 1/3: GIVEN A CoordinationGroupExists [isArchived: "False"]... Fi=001,Fa=000,Pe=001 # 2> STEP 1/3: GIVEN A CoordinationGroupExists [isArchived: "False"] (Passed after 474ms) Fi=001,Fa=000,Pe=001 # 2> STEP 2/3: WHEN Query Is Invoked [showArchived: "False"]... Fi=001,Fa=000,Pe=001 # 2> STEP 2/3: WHEN Query Is Invoked [showArchived: "False"] (Passed after 145ms) Fi=001,Fa=000,Pe=001 # 2> STEP 3/3: THEN QueryResult ShouldBeAccurate... Fi=001,Fa=000,Pe=001 # 2> STEP 3/3: THEN QueryResult ShouldBeAccurate (Passed after 5ms) Fi=002,Fa=000,Pe=000 # 2> SCENARIO RESULT: Passed after 664ms After Start Before Start Fi=002,Fa=000,Pe=001 # 3> SCENARIO: Validates QueryResults WhenDataIsSetup AndShowArchived [showArchived: "True"] Fi=002,Fa=000,Pe=001 # 3> STEP 1/3: GIVEN A CoordinationGroupExists [isArchived: "True"]... After End Fi=002,Fa=000,Pe=001 # 3> STEP 1/3: GIVEN A CoordinationGroupExists [isArchived: "True"] (Passed after 363ms) Fi=002,Fa=000,Pe=001 # 3> STEP 2/3: WHEN Query Is Invoked [showArchived: "True"]... Fi=002,Fa=000,Pe=001 # 3> STEP 2/3: WHEN Query Is Invoked [showArchived: "True"] (Passed after 64ms) Fi=002,Fa=000,Pe=001 # 3> STEP 3/3: THEN QueryResult ShouldBeAccurate... Fi=002,Fa=000,Pe=001 # 3> STEP 3/3: THEN QueryResult ShouldBeAccurate (Passed after <1ms) Fi=003,Fa=000,Pe=000 # 3> SCENARIO RESULT: Passed after 455ms After Start [testcontainers.org 00:00:39.04] Delete Docker container 2c41df1aab76 <!---- InitializeAsync Called [testcontainers.org 00:00:40.76] Docker container 92aedef3a25f created [testcontainers.org 00:00:40.81] Start Docker container 92aedef3a25f [testcontainers.org 00:00:41.57] Wait for Docker container 92aedef3a25f to complete readiness checks [testcontainers.org 00:00:41.62] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 92aedef3a25f [testcontainers.org 00:04:14.52] Execute "/bin/sh -c true && (grep -i ':01538' /proc/net/tcp || nc -vz -w 1 localhost 5432 || /bin/bash -c '</dev/tcp/localhost/5432')" at Docker container 92aedef3a25f

Passed! - Failed: 0, Passed: 3, Skipped: 0, Total: 3, Duration: 15 s - xxx.Coordination.IntegrationTests.dll (net7.0)

total of 16 tests in test project

minoseah629 commented 11 months ago

interesting.

removing [assembly: CollectionBehavior(DisableTestParallelization = true, MaxParallelThreads = 2)] got me working again. i thought i saw something to ensure 1 process is running database connections.

MeikelLP commented 11 months ago

I'm having the same issue. Sadly I don't get any logs to post here. The Ryuk container does start (logs exist) but it fails to start my custom container (Typesense). It takes about 60s until the StartAsync is finished and then I get an exception: 

 Unhandled exception. System.InvalidOperationException: Could not find resource 'TypesenseContainer'. Please create the resource by calling StartAsync(CancellationToken) or CreateAsync(CancellationToken).
brickhub-backend-1  |    at DotNet.Testcontainers.Guard.ThrowIf[TType](ArgumentInfo`1& argument, Func`2 condition, Func`2 ifClause)
brickhub-backend-1  |    at DotNet.Testcontainers.Resource.ThrowIfResourceNotFound()
brickhub-backend-1  |    at DotNet.Testcontainers.Containers.DockerContainer.GetMappedPublicPort(String containerPort)
brickhub-backend-1  |    at DotNet.Testcontainers.Containers.DockerContainer.GetMappedPublicPort(Int32 containerPort)

My setup is a little more complex, I try to start the testcontainer from within another container in a docker compose. Yes I did mount the volume for docker:

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Funnily enough this issue does not occur when running the app natively (not from within the docker container). I also noticed that the container is never created - so this is not an issue with the WaitStrategy

Any way you can help me or for me to debug this behavior?

HofmeisterAn commented 11 months ago

This looks like a Compose configuration. Did you set the environment variable mentioned here: https://dotnet.testcontainers.org/examples/dind/#compose? The Wormhole configuration varies a bit for each environment. I need more information about your setup if it is still not working.

MeikelLP commented 11 months ago

@HofmeisterAn thank you! I missed that part...