search

testcontainers / testcontainers-go

Testcontainers for Go is a Go package that makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.
https://golang.testcontainers.org
MIT License
3.36k stars 460 forks source link

[Bug]: TestGenericReusableContainerInSubprocess test is flaky #2348

Closed stevenh closed 1 week ago

stevenh commented 4 months ago

Testcontainers version

fe0d3a835bfc3183c569457a0a166f496ccbb900

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

amd64

Go version

1.22

Docker version

Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 70
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.47GiB
 Name: docker-desktop
 ID: f764de9e-312d-401f-946e-70f259d88c3a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

What happened?

Running TestGenericReusableContainerInSubprocess randomly fails.

go test -run=TestGenericReusableContainerInSubprocess -count=10 -v -failfast

Relevant log output

generic_test.go:151: 
                Error Trace:    /code/github.com/testcontainers/testcontainers-go/generic_test.go:151
                                                        /code/github.com/testcontainers/testcontainers-go/generic_test.go:135
                                                        /usr/local/go/src/runtime/asm_amd64.s:1695
                Error:          Received unexpected error:
                                exit status 1
                Test:           TestGenericReusableContainerInSubprocess
                Messages:       2024/03/13 12:46:44 github.com/testcontainers/testcontainers-go - Connected to docker: 
                                  Server Version: 25.0.3
                                  API Version: 1.44
                                  Operating System: Docker Desktop
                                  Total Memory: 15836 MB
                                  Resolved Docker Host: unix:///var/run/docker.sock
                                  Resolved Docker Socket Path: /var/run/docker.sock
                                  Test SessionID: a858391a3ac43c8efc844147a64c3b2da187684dcb4fd6294fd4ede54b3145a7
                                  Test ProcessID: ded21f82-eb8e-4f3d-b7c8-0b76f0a1e86b
                                2024/03/13 12:46:45 🔥 Reaper obtained from Docker for this test session 0c297d9c28fc1fb90c56227e03f6a2a141389b695c79ef143edfdd7bd54bc66f
                                2024/03/13 12:46:45 Connect: a858391a3ac43c8efc844147a64c3b2da187684dcb4fd6294fd4ede54b3145a7
                                2024/03/13 12:46:45 ✅ Container started: c392e3a78ec5
                                2024/03/13 12:46:45 🚧 Waiting for container id c392e3a78ec5 image: docker.io/menedev/delayed-nginx:1.15.2. Waiting for: &{Port:80/tcp timeout:<nil> PollInterval:100ms}
                                2024/03/13 12:46:45 🔔 Container is ready: c392e3a78ec5
                                ContainerID: c392e3a78ec50a46fbf67d7d17c725582ccbc4cb0377bd54d470ba08c21b1e25
                                --- FAIL: TestHelperContainerStarterProcess (0.95s)
                                    generic_test.go:191: 
                                                Error Trace:    /code/github.com/testcontainers/testcontainers-go/generic_test.go:191
                                                Error:          Received unexpected error:
                                                                Get "http://localhost:57916": EOF
                                                Test:           TestHelperContainerStarterProcess
                                FAIL

Additional information

This is a bug in the reaper, PR incoming.

stevenh commented 4 months ago

PR is now open against moby-ryuk which fixes the underlying issue which is that the reaper container doesn't reset it's waiting for connection timeout, which means it will randomly shutdown while in use if its connection count ever reaches zero.

Once that's merged and a new release made, we'll need to PR to bump the container image version.

stevenh commented 4 months ago

Testing locally I can now run 100 iterations without fail, where as previously 10 was usually enough to trigger an issue.

go test -run=TestGenericReusableContainerInSubprocess -count=100 -v -failfast
mdelapenya commented 4 months ago

Super deep investigation, thanks for doing this!

stevenh commented 4 months ago

No problem @mdelapenya just been tripping over issues trying to get a clean test run, so now have a few PR's in flight.

Looks like someone authorised the test run which has now passed, don't suppose you have the power to do a merge and release so I can get the image PR done do you as this is going to be causing all sorts of issues so would be good to get it in?

stevenh commented 3 months ago

Should we re-open until we have a release and PR to bump the container version?

mdelapenya commented 3 months ago

Reopening until the next release of Ryuk, which will solve this issue

stevenh commented 3 months ago

@mdelapenya not sure if you intended to close this with the PR against a fork of moby-ryuk?

mdelapenya commented 3 months ago

@mdelapenya not sure if you intended to close this with the PR against a fork of moby-ryuk?

Oh my! I pushed to my origin right after updating ryuk and it seems it triggered the auto-close. Reopening 🤦

stevenh commented 2 weeks ago

For those seeing this try cloning the moby-ryuk repo and running the following in it to replace the image that testcontainers-go uses to see if it does fix:

docker build -f linux/Dockerfile -t testcontainers/ryuk:0.7.0 .
mdelapenya commented 1 week ago

Given ryuk:0.8.1 was released, and the main branch comes with it, I think we can close this one.

Please reopen if you consider it's not fixed, thanks!