search

testcontainers / testcontainers-go

Testcontainers for Go is a Go package that makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.
https://golang.testcontainers.org
MIT License
3.69k stars 507 forks source link

[Bug]: Could not start container: creating reaper failed #2477

Open sebastianbuechler opened 7 months ago

sebastianbuechler commented 7 months ago

Testcontainers version

0.30.0

Using the latest Testcontainers version?

Yes

Host OS

Windows 11

Host arch

amd64

Go version

1.21.6

Docker version

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:02 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

$ docker info
Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 44
  Running: 0
  Paused: 0
  Stopped: 44
 Images: 22
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 15.58GiB
 Name: docker-desktop
 ID: d6a66196-edce-47b4-86af-8679687c5b0c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?

I ran an integration test that spins up a PostgreSQL database which usually works without any issue on other systems. But on this machine it says:

2024/04/10 13:59:43 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 25.0.3
  API Version: 1.44
  Operating System: Docker Desktop
  Total Memory: 15951 MB
  Resolved Docker Host: tcp://127.0.0.1:50934
  Resolved Docker Socket Path: //./pipe/docker_engine
  Test SessionID: 5fa5f11f9fba7e7d7f5af0266e5babf8bc7030bec98b3d7607c4f7fd6ae973bb
  Test ProcessID: e97481d1-d819-42e3-bc0f-f07aad00f7a0
2024/04/10 13:59:43 Failed to get image auth for https://index.docker.io/v1/. Setting empty credentials for the image: testcontainers/ryuk:0.7.0. Error is:credentials not found in native keychain
2024/04/10 13:59:46 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/04/10 13:59:46 ✅ Container created: 9433fe236771
2024/04/10 13:59:46 🐳 Starting container: 9433fe236771
2024/04/10 13:59:46 ✅ Container started: 9433fe236771
2024/04/10 13:59:46 🚧 Waiting for container id 9433fe236771 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/04/10 13:59:47 failed accessing container logs: Error response from daemon: No such container: 9433fe2367714ff9aa681c31b776f9ceead5f282e8413c846641de31afd6e5ea
2024/04/10 13:59:47 Error response from daemon: No such container: 9433fe2367714ff9aa681c31b776f9ceead5f282e8413c846641de31afd6e5ea: could not start container: creating reaper failed: failed to create container

Relevant log output

No response

Additional information

No response

dmakushin commented 7 months ago

Same for me. Started after update to 0.30.0

weeco commented 7 months ago

V0.30.0 has been causing issues in my team as well. Different setups running Docker Desktop and Rancher - both running consistently into:

port not found: creating reaper failed: failed to create container

or

dial tcp [::1]:32810: connect: connection refused: Connecting to Ryuk on localhost:32810 failed: connecting to network reaper failed: failed to create network

Downgrading to v0.29.2 fixed it.

mdelapenya commented 7 months ago

Mmm I'm going to prioritise this, as it should work.... Please check the +100 usages of this new release without issue (I supose) https://github.com/search?q=%22testcontainers-go+v0.30%22+path%3Ago.mod+NOT+is%3Afork+NOT+org%3Atestcontainers+&type=code

From the logs, it seems Ryuk dies before the next container is started. Could you try enabling the verbose mode in Ryuk and trying to capture Ryuk logs and see if the connections are happening?

miparnisari commented 7 months ago

👋 seeing the same issue. https://github.com/openfga/openfga/actions/runs/8926586012/job/24517945125.

stubents commented 6 months ago

I'm not sure about the original error in this issue but the "port not found: creating reaper failed: failed to create container" that was mentioned by @weeco seems to be related to https://github.com/testcontainers/testcontainers-go/issues/605

With the commit 5fa65489 DockerContainer.MappedPort(...) now uses the ContainerJSON cache in Inspect() instead of always going for the client. And sometimes that cache happens to miss the mapping in the network settings.

In my setup this doesn't only affect Ryuk but other containers as well. Commenting out the early return at https://github.com/testcontainers/testcontainers-go/blob/5fa65489ac33e7a6a15e4599739b2c717cf3a1a9/docker.go#L168 solved the issue for me, but I'm not sure if this is a valid solution :)

pr0ton11 commented 6 months ago

I encountered the same issue. Workaround of @stubents did not fix it for me.

stubents commented 6 months ago

I encountered the same issue. Workaround of @stubents did not fix it for me.

Now, that I downgraded to v0.29.1 I still see those "port not found" errors coming every now and then, so I guess 5fa6548 just made them more likely to happen.

kiview commented 6 months ago

I guess this should be fixed by https://github.com/testcontainers/testcontainers-go/pull/2508.

pcriv commented 5 months ago

@kiview I'm still having this issue with colima version 0.6.9, is there anything I can do to fix that?

vec715 commented 5 months ago

any updates?

stevenh commented 4 months ago

Could be related to the fix here https://github.com/testcontainers/moby-ryuk/pull/121, still needs a release and then testcontainers-go updating to use the new image.

Try cloning the moby-ryuk repo and running the following in it to replace the image that testcontainers-go uses to see if it does fix:

docker build -f linux/Dockerfile -t testcontainers/ryuk:0.7.0 .
strowk commented 3 months ago

Just got this with version 0.32.0 on my Ubuntu machine. We get this once in awhile on CI as well (Github Actions), but I haven't seen this happening often on dev machine yet.

panic: Error response from daemon: No such container: e4e95c74d1f499df597b3f93b8d3bb1df5bcfc11ca64cb63ac38c08607d50380: creating reaper failed: failed to create container [recovered] panic: Error response from daemon: No such container: e4e95c74d1f499df597b3f93b8d3bb1df5bcfc11ca64cb63ac38c08607d50380: creating reaper failed: failed to create container

Feels like, this should not be a closed issue, as it still happens on latest published version?

mdelapenya commented 3 months ago

@strowk does it happen using the HEAD commit in main?

We usually close the issues when PR fixing them are merged (thanks to Github links) so I think it's a common practice to do that.

pixelsoccupied commented 1 month ago

Hello! Trying out testcontainer for the first time and just into this issue. I see the bump Ryuk 0.8.1 but still getting this error

create container: started hook: wait until ready: dial tcp: lookup null: no such host: could not start container: creating reaper failed

My setup is super minimal. Just want to bring up a postgres container. 

pgContainer, err := postgres.Run(ctx,
        "postgres:16-alpine",
        postgres.WithDatabase("test-db"),
        postgres.WithUsername("postgres"),
        postgres.WithPassword("postgres"),
        testcontainers.WithWaitStrategy(
            wait.ForLog("database system is ready to accept connections").
                WithOccurrence(2).WithStartupTimeout(5*time.Second)),
    )

Using Colima + macOS. Curious if there's any workaround for this?

pixelsoccupied commented 1 month ago

Ah as a workaround I set env var TESTCONTAINERS_RYUK_DISABLED to true. It seems to get things going!

Related issue i think https://github.com/testcontainers/testcontainers-go/issues/2798 ?

marcosfilhote commented 1 month ago

Same problem here: 

pgContainer, err := containerPostgres.Run(ctx,
        "docker.io/postgres:13.3-alpine",
        // containerPostgres.WithInitScripts(filepath.Join("../../assets", "db", "initdb.sql")),
        // containerPostgres.WithConfigFile(filepath.Join("testdata", "my-postgres.conf")),
        containerPostgres.WithDatabase(dbName),
        containerPostgres.WithUsername(dbUser),
        containerPostgres.WithPassword(dbPassword),
        // testcontainers.WithHostPortAccess(5432),
        testcontainers.WithWaitStrategy(
            wait.ForLog("database system is ready to accept connections").
                WithStartupTimeout(25*time.Second),
            wait.ForListeningPort("5432/tcp"),
        ),
    )

setting TESTCONTAINERS_RYUK_DISABLED to true to keep working.

stevenh commented 1 month ago

Could you check with the branch from https://github.com/testcontainers/testcontainers-go/pull/2728 to see if that fixes the issue for you @marcosfilhote please?