Closed abemedia closed 1 week ago
Hi @abemedia ! I hope you are doing well! can you share link to the job? I check actions in this repo - repear works fine - for example - https://github.com/testcontainers/testcontainers-go/actions/runs/7730045562/job/21075148438#step:10:41
Did you manage to solve this issue? I'm facing an issue with reaper failing with the following error.. it only happens on GHA:
2024/02/13 14:47:24 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 25.0.3
API Version: 1.43
Operating System: Alpine Linux v3.19 (containerized)
Total Memory: 31291 MB
Resolved Docker Host: unix:///run/docker.sock
Resolved Docker Socket Path: /run/docker.sock
Test SessionID: c00c76c83c7180c78797a94d2146b5216f91200d033f9eb87efd620b53b27115
Test ProcessID: e60189dd-ddfc-4b4c-98ab-1[70](xxx)99ea7a937
2024/02/13 14:47:24 π³ Creating container for image testcontainers/ryuk:0.6.0
internal_handler_test.go:283: failed to start container: look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container
Did you manage to solve this issue? I'm facing an issue with reaper failing with the following error.. it only happens on GHA:
2024/02/13 14:47:24 github.com/testcontainers/testcontainers-go - Connected to docker: Server Version: 25.0.3 API Version: 1.43 Operating System: Alpine Linux v3.19 (containerized) Total Memory: 31291 MB Resolved Docker Host: unix:///run/docker.sock Resolved Docker Socket Path: /run/docker.sock Test SessionID: c00c76c83c7180c78797a94d2146b5216f91200d033f9eb87efd620b53b27115 Test ProcessID: e60189dd-ddfc-4b4c-98ab-1[70](xxx)99ea7a937 2024/02/13 14:47:24 π³ Creating container for image testcontainers/ryuk:0.6.0 internal_handler_test.go:283: failed to start container: look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container
I am getting the same error apparently.
@kostyay I was able to bypass this error by cleaning up containers on the tests and then using TESTCONTAINERS_RYUK_DISABLED: true
. FYI I realize this happens mostly when there are multiple tests spawning containers.
I am experiencing this exact same issue in gitlab runners when running a few different jobs that are all using testcontainers. 4/5 of my jobs always pass and one job (which runs tests in parallel) always fails when the other 4 are running but passes when I retry it (and it is running by itself).
I am getting a similar error in Github Actions:
Server Version: 25.0.2
API Version: 1.44
Operating System: Ubuntu 20.04.6 LTS
Total Memory: 7751 MB
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
2024/03/19 13:49:59 π³ Creating container for image testcontainers/ryuk:0.6.0
Error: Received unexpected error:
look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container
I've seen this quite frequently in Jenkins when running a large number of tests. Disabling the reaper with TESTCONTAINERS_RYUK_DISABLED
fixed the issue for me.
If you're in an ephemeral environment enabling TESTCONTAINERS_RYUK_DISABLED
should be a fine workaround without any negative consequences.
Adding a data point here: we never hit this until upgrading testcontainers-go from 0.19 to 0.28. \ Soon after, starting hitting this issue fairly regularly (I would estimate about 1 out of 4 jobs fail due to either
look up reaper container returned nil although creation failed due to name conflict
or
connecting to reaper failed: failed to create container
Still having this issue. I notice this when multiple containers are spawned. But when done individually, no errors of reaper or port issues occurred. Having the (cached)
text makes the test pass.
--- PASS: TestProducts (3.18s)
--- PASS: TestProducts/getting_product_of_23 (0.00s)
--- PASS: TestProducts/getting_product_of_14 (0.00s)
--- PASS: TestProducts/getting_product_of_4fh555sd (0.00s)
--- PASS: TestProducts/getting_product_of_-23.42 (0.00s)
--- PASS: TestProducts/getting_product_of_234234234 (0.00s)
--- PASS: TestProducts/creating_product_case_index:_0 (0.36s)
--- PASS: TestProducts/creating_product_case_index:_1 (0.25s)
--- PASS: TestProducts/creating_product_case_index:_2 (0.25s)
PASS
ok github.com/bhlox/ecom/internal/services/product (cached)
To narrow @tuxxi 's anecdote. I didn't see this until we upgraded from 0.26 to 0.27
which indirectly changes the ryuk image version as well.
Downgrading back is not an option because we did it to be able to use TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX
Hi folks, I think in the most recent versions we have introduced some fixes to the synchronisation of the reaper over a single test session. Issues/PRs of interest could be:
I'd appreciate if you can check them out so we can close this one.
Thanks in advance for your support!
@mdelapenya Just checked on Windows 11 with WSL2 docker. Test container version: v0.31.0. Had one error: "port not found: creating reaper failed: failed to create container".
TL;DR: You can terminate the containers directly and no longer need the reaper.
Hey, so I was having this issue. I'm using an old version of the project but I think the solution should work for newer versions.
On my ContainerRequest
I added SkipReaper: true
so that the ryuk reaper does not start up. Then I have some code that gets the created Container
passed to it and uses it for my tests and that looks like this:
func TestMain(m *testing.M) {
container, testDB, err := tools.CreateContainer("test-db")
if err != nil {
log.Fatal(err)
}
TestSqlDB = testDB
result := m.Run()
// Deferred functions aren't run when os.Exit is called..
testDB.Close()
container.Terminate(context.Background()) // nolint
os.Exit(result)
}
My issue was originally that the container.Terminate
was not running because it was deferred and the os.Exit
was preventing the deferred functions from executing. This meant I had to depend on the reaper to clear the created containers. I only realized this once Github Actions started failing my builds. With skipping the reaper and fixing this deferred termination of the container everything runs better than it even did before. Containers start and die super fast.
I added SkipReaper: true so that the ryuk reaper does not start up
For the record, SkipReaper is deprecated in newer versions and will eventually disappear from the Request struct (see v1 branch: https://github.com/testcontainers/testcontainers-go/blob/v1/request.go#L30)
If you want to disable Ryuk, please use the properties file here: https://golang.testcontainers.org/features/configuration/#customizing-ryuk-the-resource-reaper
I'm also experiencing this in a regular dev environment (it's not a GitHub Action).
It would be great if it just worked with Ryuk without needing to turn it off and manually manage the container state. I don't think this is a good solution. Ryuk should just work out of the box.
This is especially important because it's challenging to manage container state across many tests run in parallel if you want to reuse containers (instead of creating a new container for each test). I'd much rather have Ryuk handle this itself, so I can run my tests in parallel without having to manage logic for terminating the containers once all tests have finished.
Hopefully this bug gets fixed soon because it's been a PITA.
Possibly related to this fix https://github.com/testcontainers/moby-ryuk/pull/121 which needs a release and then testcontainers-go updating to use the new image version
Try cloning the moby-ryuk repo and running the following in it to replace the image that testcontainers-go uses to see if it does fix:
docker build -f linux/Dockerfile -t testcontainers/ryuk:0.7.0 .
You can terminate the containers directly and no longer need the reaper.
@tbrown1979 please remember that the reaper also removes built images, volumes and networks, so if you disable it please carefully remove all those Docker resources.
In any case, we have released a new version of Ryuk. As @stevenh pointed out in https://github.com/testcontainers/testcontainers-go/issues/2172#issuecomment-2226558170, you could give it a try like that.
Testcontainers version
v0.27.0
Using the latest Testcontainers version?
Yes
Host OS
Ubuntu 22.04.3 LTS
Host arch
amd64
Go version
1.20
Docker version
Docker info
What happened?
This does not seem to happen when running locally but occasionally my tests fail in GitHub Actions with the error
creating reaper failed
.Relevant log output
Additional information
No response