testcontainer-go: Could not setup Async I/O: 65536

[roarc0]

Version & Environment

Redpanda version: 23.3.3 or any other version

What went wrong?

I have a go code base that is currently using testcontainers 0.13, I'm trying to update to new version of that tool. Many tests are using custom test redpanda containers. The problem is that in CI (circleci) I get this error: Could not setup Async I/O: 65536. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr (Resource temporarily unavailable)

What should have happened instead?

The container should start without errors.

How to reproduce the issue?

try to create a custom redpanda testcontainer as follows:

    container, err := testcontainers.GenericContainer(
        ctx,
        testcontainers.GenericContainerRequest{
            ContainerRequest: testcontainers.ContainerRequest{
                Name:  name,
                Image: "docker.redpanda.com/redpandadata/redpanda:v23.3.3",
                User:  "root:root",
                ExposedPorts: []string{
                    fmt.Sprintf("%d:9092/tcp", freePort),
                    "9644/tcp",
                },
                Cmd: []string{
                    "redpanda",
                    "start",
                    "--mode=dev-container",
                    "--smp=1",
                    "--memory=1G",
                    fmt.Sprintf("--advertise-kafka-addr=localhost:%d", freePort),
                },
                WaitingFor: wait.ForLog("Successfully started Redpanda!").WithStartupTimeout(2 * time.Minute),
            },
            Logger:  logger,
            Started: true,
        },
    )

Additional information

2024/06/20 11:26:49 🔥 Reaper obtained from Docker for this test session 4868c35e6f020c2485f0d4fe93748e5c35f3582b7d905675079623404a1751e5
{"level":"debug","ts":1718882809.8739932,"logger":"root","caller":"testcontainers-go/lifecycle.go:60","msg":"🐳 Creating container for image docker.redpanda.com/redpandadata/redpanda:v23.3.3"}
{"level":"debug","ts":1718882809.937674,"logger":"root","caller":"testcontainers-go/lifecycle.go:66","msg":"✅ Container created: 2e34e10495a0"}
{"level":"debug","ts":1718882809.9377205,"logger":"root","caller":"testcontainers-go/lifecycle.go:72","msg":"🐳 Starting container: 2e34e10495a0"}
{"level":"debug","ts":1718882810.1308205,"logger":"root","caller":"testcontainers-go/lifecycle.go:78","msg":"✅ Container started: 2e34e10495a0"}
{"level":"debug","ts":1718882810.13087,"logger":"root","caller":"testcontainers-go/lifecycle.go:193","msg":"🚧 Waiting for container id 2e34e10495a0 image: docker.redpanda.com/redpandadata/redpanda:v23.3.3. Waiting for: &{timeout:0xc0004f96e8 Log:Successfully started Redpanda! IsRegexp:false Occurrence:1 PollInterval:100ms}"}
{"level":"debug","ts":1718882810.4999342,"logger":"root","caller":"testcontainers-go/lifecycle.go:288","msg":"container logs (container exited with code 139):\n+ '[' '' = true ']'\n+ exec /usr/bin/rpk redpanda start --mode=dev-container --smp=1 --memory=1G --advertise-kafka-addr=localhost:33081\nWARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly.\nWe'd love to hear about your experience with Redpanda:\nhttps://redpanda.com/feedback\nStarting redpanda...\nRunning:\n/opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --reserve-memory=0M --overprovisioned --lock-memory=false --smp=1 --unsafe-bypass-fsync=true --memory=1G\nlibc++abi: terminating due to uncaught exception of type std::runtime_error: Could not setup Async I/O: 65536. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr (Resource temporarily unavailable). Try increasing that number or reducing the amount of logical CPUs available for your application"}
--- FAIL: TestAdServerRequestToPostgresAdUnitCodeTable (0.64s)
    start_test.go:33: 
                Error Trace:    /home/circleci/project/service/inventory-rollup-ad-unit-codes/start_test.go:33
                Error:          Received unexpected error:
                                start container: container exited with code 139
                Test:           TestAdServerRequestToPostgresAdUnitCodeTable
                Messages:       create redpanda container
FAIL

JIRA Link: CORE-4261

[vsarunas]

I ran into this log as well; try to double on the host which runs testcontainer:

sudo sysctl -w fs.aio-max-nr=1048576

[roarc0]

thanks, I tried this again (didn't work before) directly on the CI and it works.