Multiple "regression" on 10.7.0 (wait strategy) and 10.8.0 (image substitution) when running using kubedock environment

[jonesbusy]

Hi,

I already posted some details on https://github.com/joyrex2001/kubedock/issues/84 because I'm not sure where is located the issue.

We noticed behavior/regression changes when upgrading from 10.6 to 10.8.1 when running using kubedock on Kubernetes

We have the same setup with Testcontainer java (version 1.19.7) and testcontainer .NET (version 3.7.0) and all works fine

But for some reason node module is not working post 10.6

I noticied 2 PR that could have cause such changes

https://github.com/testcontainers/testcontainers-node/pull/712 (released on 10.7) https://github.com/testcontainers/testcontainers-node/pull/729 (release on 10.8)

The issue is not producible using a "real" docker socket with Rancher Deskop

Expected Behaviour

Should work

Actual Behaviour

Container is started like this

    wireMockContainer = await new GenericContainer("wiremock/wiremock:3.3.1-2")
      .withExposedPorts(8080)
      .withCommand(["--verbose"])
      .withWaitStrategy(Wait.forLogMessage(/port:.*8080/))
      .withStartupTimeout(120000)
      .start();

On 10.7

[11:33:21.886+02:00] - 2024-04-03T09:33:21.059Z testcontainers [ERROR] [283b40bf8ce7] Container failed to be ready: Error: Log stream ended and message "/.+ Started!/" was not received

[11:33:21.885+02:00] - 2024-04-03T09:33:20.978Z testcontainers [DEBUG] [283b40bf8ce7] Waiting for container to be ready...
[11:33:21.885+02:00] - 2024-04-03T09:33:20.979Z testcontainers [DEBUG] [283b40bf8ce7] Waiting for log message "/.+ Started!/"...
[11:33:21.885+02:00] - 2024-04-03T09:33:20.979Z testcontainers [DEBUG] [283b40bf8ce7] Fetching container logs...
[11:33:21.885+02:00] - 2024-04-03T09:33:20.982Z testcontainers [DEBUG] [283b40bf8ce7] Demuxing stream...
[11:33:21.885+02:00] - 2024-04-03T09:33:20.982Z testcontainers [DEBUG] [283b40bf8ce7] Demuxed stream
[11:33:21.885+02:00] - 2024-04-03T09:33:20.982Z testcontainers [DEBUG] [283b40bf8ce7] Fetched container logs
[11:33:21.886+02:00] - 2024-04-03T09:33:21.058Z testcontainers [ERROR] [283b40bf8ce7] Log stream ended and message "/.+ Started!/" was not received
[11:33:21.886+02:00] - 2024-04-03T09:33:21.059Z testcontainers [ERROR] [283b40bf8ce7] Container failed to be ready: Error: Log stream ended and message "/.+ Started!/" was not received
[11:33:21.886+02:00] - 2024-04-03T09:33:21.059Z testcontainers [DEBUG] [283b40bf8ce7] Stopping container...
[11:33:21.886+02:00] - 2024-04-03T09:33:21.202Z testcontainers [DEBUG] [283b40bf8ce7] Stopped container
[11:33:21.886+02:00] - 2024-04-03T09:33:21.202Z testcontainers [DEBUG] [283b40bf8ce7] Removing container...
[11:33:21.886+02:00] - 2024-04-03T09:33:21.205Z testcontainers [DEBUG] [283b40bf8ce7] Removed container
[11:33:21.886+02:00] - 2024-04-03T09:33:21.205Z testcontainers [DEBUG] Releasing lock file "/tmp/testcontainers-node.lock"...
[11:33:21.886+02:00] - 2024-04-03T09:33:21.206Z testcontainers [DEBUG] Released lock file "/tmp/testcontainers-node.lock"

This was working fine on 10.6

On 10.8.1

Basically it hangs on Acquiring lock file

[11:23:05.306+02:00] - 2024-04-03T09:23:05.264Z testcontainers [DEBUG] Acquired lock file "/tmp/testcontainers-node.lock"
[11:23:05.306+02:00] - 2024-04-03T09:23:05.264Z testcontainers [DEBUG] Listing containers...
[11:23:05.306+02:00] - 2024-04-03T09:23:05.267Z testcontainers [DEBUG] Listed containers
[11:23:05.306+02:00] - 2024-04-03T09:23:05.268Z testcontainers [DEBUG] Creating new Reaper for session "4985d02b214e" with socket path "/var/run/docker.sock"...
[11:23:05.306+02:00] - 2024-04-03T09:23:05.268Z testcontainers [INFO] Applying changes to image testcontainers/ryuk with tag 0.5.1: added registry r-docker-registry-1-docker-io.artifactory.******/
[11:23:05.306+02:00] - 2024-04-03T09:23:05.268Z testcontainers [DEBUG] Checking if image exists "r-docker-registry-1-docker-io.artifactory********//testcontainers/ryuk:0.5.1"...
[11:23:05.306+02:00] - 2024-04-03T09:23:05.270Z testcontainers [DEBUG] Checked if image exists "r-docker-registry-1-docker-io.artifactory*******//testcontainers/ryuk:0.5.1"
[11:23:05.306+02:00] - 2024-04-03T09:23:05.270Z testcontainers [DEBUG] Image "r-docker-registry-1-docker-io.artifactory.******//testcontainers/ryuk:0.5.1" already exists
[11:23:05.306+02:00] - 2024-04-03T09:23:05.270Z testcontainers [DEBUG] Acquiring lock file "/tmp/testcontainers-node.lock"...
[11:24:41.996+02:00] -  ❯ src/ts/api.test.ts  (5 tests) 90133ms
[11:24:41.996+02:00] - 
[11:24:41.996+02:00] - ⎯⎯⎯⎯⎯⎯ Failed Suites 1 ⎯⎯⎯⎯⎯⎯⎯
[11:24:41.996+02:00] - 
[11:24:41.996+02:00] -  FAIL  src/ts/api.test.ts > api
[11:24:41.996+02:00] - Error: Hook timed out in 90000ms.

Since 10.6 I was using environment (but didn't realized it was not used before https://github.com/testcontainers/testcontainers-node/pull/729)

TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX  = 'r-docker-registry-1-docker-io.artifactory.*********/'

Noticed the double // on the logs. I tried also with no trailing slash and behavior is the same.

Any hints is appreciated!

Thanks!

I'm more happy to give more detail/logs if needed

[ilia-beliaev-miro]

Made a PR with the fix here - https://github.com/testcontainers/testcontainers-node/pull/748

[joyrex2001]

I don't think this will fix the problem. The root-cause for https://github.com/joyrex2001/kubedock/issues/84 is this constant being compared with a rewritten image. Because the isReaper detection fails, it tries to create another lockfile (which fails, because it is locked already).

My suggestion would be to prefix the default value for REAPER_IMAGE with TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX environment variable (but there might be a better solution).

To reproduce: set TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=docker.io and it will fail starting ryuk (and trigger the dead-lock).

[cristianrgreco]

Please try version 10.8.2

[ilia-beliaev-miro]

Tried with our proxy and env variable having a form export TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=domain.com/docker-virtual/ and it worked.

[cristianrgreco]

Will give some time for @jonesbusy or @joyrex2001 to confirm, and then will close this issue.

[jonesbusy]

Thanks, I will try to test tomorrow

[jonesbusy]

Hi,

Thanks again for the fix and help of @joyrex2001

I confirm issues I was facing are fixed with 10.8.2

Regards,