search

testcontainers / testcontainers-go

Testcontainers for Go is a Go package that makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.
https://golang.testcontainers.org
MIT License
3.64k stars 500 forks source link

[Bug]: create container: port not found: creating reaper failed #2798

Open mukulmantosh opened 1 month ago

mukulmantosh commented 1 month ago

Testcontainers version

0.33.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x86

Go version

1.23

Docker version

Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:17:17 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.34.2 (167172)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:15 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.2.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 35
  Running: 19
  Paused: 0
  Stopped: 16
 Images: 19
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 31.24GiB
 Name: docker-desktop
 ID: d257b354-0307-46dc-98db-5e5637a2026d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?

I attempted to run tests that involve both Postgres and NATS. While they often run smoothly on their own, there are times when they fail unexpectedly.

I am not using the GenericContainer, instead using the official supported ones.

Example: Postgres

ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()

pgContainer, err = postgres.Run(ctx, containerImage,
    postgres.WithDatabase(dbName),
    postgres.WithUsername(dbUsername),
    postgres.WithPassword(dbPassword),
    testcontainers.WithWaitStrategy(
        wait.ForLog("database system is ready to accept connections").
            WithOccurrence(2).WithStartupTimeout(10*time.Second)),
)
if err != nil {
    log.Fatalf("failed to start PostgreSQL container: %s", err)
}

// Set environment variables
dbHost, _ = pgContainer.Host(ctx)
mappedPort, _ := pgContainer.MappedPort(ctx, "5432/tcp")
dbPort = mappedPort.Port()
_ = os.Setenv("DB_USERNAME", dbUsername)
_ = os.Setenv("DB_PASSWORD", dbPassword)
_ = os.Setenv("DB_NAME", dbName)
_ = os.Setenv("DB_PORT", dbPort)
_ = os.Setenv("DB_HOST", dbHost)

NATS

natsContainer, err := nats.Run(ctx, "nats:2.10.20", testcontainers.WithHostPortAccess(4222))

if err != nil {
    t.Logf("failed to start NATS container: %s", err)
    return
}

Relevant log output

failed to start NATS container: create container: port not found: creating reaper failed

Additional information

No response

mukulmantosh commented 1 month ago

go.mod 

github.com/testcontainers/testcontainers-go v0.33.0
github.com/testcontainers/testcontainers-go/modules/nats v0.33.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.32.0
stevenh commented 1 month ago

There are know race conditions in the reaper which should be fixed once we've merged https://github.com/testcontainers/testcontainers-go/pull/2728 and updated to use the new reaper image container the work which was merged today https://github.com/testcontainers/moby-ryuk/pull/141

erikmansson commented 4 weeks ago

@mukulmantosh did you find any workaround for this issue?

stevenh commented 4 weeks ago

If you use the branch from https://github.com/testcontainers/testcontainers-go/pull/2728 does it fix your issue?

erikmansson commented 3 weeks ago

On the merged commit, we now instead get

            Error:          Received unexpected error:
                            generic container: create container: reaper: from container "1aabe62f": wait for reaper 1aabe62f: external check: check target: retries: 571 address: 172.19.0.1:58322: get state: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.46/containers/1aabe62f3a5988e7bf88dbae63aa756f9700f3c6697eb41fa6b5f56933bb080c/json": context deadline exceeded
stevenh commented 3 weeks ago

Can you confirm you're using v0.34.0 and the exact test case you're using?

stevenh commented 3 weeks ago

I've tried with the following and all works as expected, so suspecting it could be something setup releated

package testcontainers_test

import (
    "context"
    "testing"
    "time"

    "github.com/stretchr/testify/require"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/modules/postgres"
    "github.com/testcontainers/testcontainers-go/wait"
)

func TestReaper(t *testing.T) {
    ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()

    dbName, dbUsername, dbPassword := "test_db", "test_user", "test_password"

    pgContainer, err := postgres.Run(ctx, "docker.io/postgres:16-alpine",
        postgres.WithDatabase(dbName),
        postgres.WithUsername(dbUsername),
        postgres.WithPassword(dbPassword),
        testcontainers.WithWaitStrategy(
            wait.ForLog("database system is ready to accept connections").
                WithOccurrence(2).WithStartupTimeout(10*time.Second),
        ),
    )
    testcontainers.CleanupContainer(t, pgContainer)
    require.NoError(t, err)

    dbHost, err := pgContainer.Host(ctx)
    require.NoError(t, err)
    require.NotEmpty(t, dbHost)

    mappedPort, err := pgContainer.MappedPort(ctx, "5432/tcp")
    require.NoError(t, err)
    require.NotEmpty(t, mappedPort)
}

go.mod:

module github.com/testcontainers/testcontainers-go/test

go 1.23.2

require (
        github.com/stretchr/testify v1.9.0
        github.com/testcontainers/testcontainers-go v0.34.0
        github.com/testcontainers/testcontainers-go/modules/postgres v0.34.0
)

require (
        dario.cat/mergo v1.0.0 // indirect
        github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
        github.com/Microsoft/go-winio v0.6.2 // indirect
        github.com/cenkalti/backoff/v4 v4.2.1 // indirect
        github.com/containerd/containerd v1.7.18 // indirect
        github.com/containerd/log v0.1.0 // indirect
        github.com/containerd/platforms v0.2.1 // indirect
        github.com/cpuguy83/dockercfg v0.3.2 // indirect
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/distribution/reference v0.6.0 // indirect
        github.com/docker/docker v27.1.1+incompatible // indirect
        github.com/docker/go-connections v0.5.0 // indirect
        github.com/docker/go-units v0.5.0 // indirect
        github.com/felixge/httpsnoop v1.0.4 // indirect
        github.com/go-logr/logr v1.4.1 // indirect
        github.com/go-logr/stdr v1.2.2 // indirect
        github.com/go-ole/go-ole v1.2.6 // indirect
        github.com/gogo/protobuf v1.3.2 // indirect
        github.com/google/uuid v1.6.0 // indirect
        github.com/klauspost/compress v1.17.4 // indirect
        github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
        github.com/magiconair/properties v1.8.7 // indirect
        github.com/moby/docker-image-spec v1.3.1 // indirect
        github.com/moby/patternmatcher v0.6.0 // indirect
        github.com/moby/sys/sequential v0.5.0 // indirect
        github.com/moby/sys/user v0.1.0 // indirect
        github.com/moby/term v0.5.0 // indirect
        github.com/morikuni/aec v1.0.0 // indirect
        github.com/opencontainers/go-digest v1.0.0 // indirect
        github.com/opencontainers/image-spec v1.1.0 // indirect
        github.com/pkg/errors v0.9.1 // indirect
        github.com/pmezard/go-difflib v1.0.0 // indirect
        github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
        github.com/shirou/gopsutil/v3 v3.23.12 // indirect
        github.com/shoenig/go-m1cpu v0.1.6 // indirect
        github.com/sirupsen/logrus v1.9.3 // indirect
        github.com/tklauser/go-sysconf v0.3.12 // indirect
        github.com/tklauser/numcpus v0.6.1 // indirect
        github.com/yusufpapurcu/wmi v1.2.3 // indirect
        go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
        go.opentelemetry.io/otel v1.24.0 // indirect
        go.opentelemetry.io/otel/metric v1.24.0 // indirect
        go.opentelemetry.io/otel/trace v1.24.0 // indirect
        golang.org/x/crypto v0.24.0 // indirect
        golang.org/x/sys v0.21.0 // indirect
        gopkg.in/yaml.v3 v3.0.1 // indirect
)
erikmansson commented 2 weeks ago

We later tried v0.34.0 as well - no difference. However, we did figure out that it has something to do with running the test inside a Docker container with /var/run/docker.sock mounted. Guess that's a separate issue.

weihong-xu-hpe commented 6 days ago

We saw recently the same on github actions more and more common for this issues. Running at github codespace those issues are gone. v0.33.0 and v0.34.0 get no difference, always some random testcontainer connection failed.