[Enhancement]: Extending Testcontainers to support multiple container runtimes (Podman)

[HofmeisterAn]

Problem

Since Testcontainers for Node has done an awesome job supporting multiple container runtimes (not just Docker), it makes perfect sense to continue this great work and look into implementing it for .NET. We can start by collecting a list of tasks, issues or incompabilities that are necessary to support other runtimes as well.

• [x] Docker.DotNet's ExtractArchiveToContainerAsync(string, ContainerPathStatParameters, Stream, CancellationToken) throws System.Net.Sockets.SocketException : Connection reset by peer.
  • ✅ https://github.com/dotnet/Docker.DotNet/pull/629
• [x] Running Testcontainers for .NET on macOS using Podman (brew installation) does not work well with the Resource Reaper. Testcontainers is incapable of mounting the daemon socket. It fails with different errors:

  Docker API responded with status code=InternalServerError, response={"cause":"operation not supported","message":"container create: statfs /Users/atomicjar/.local/share/containers/podman/machine/qemu/podman.sock: operation not supported","response":500}

  When attempting to use the daemon socket path from within the Podman virtual machine, the process is also unsuccessful. Setting export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock fails:

  Docker API responded with status code=InternalServerError, response={"cause":"permission denied","message":"container create: statfs /var/run/docker.sock: permission denied","response":500}

  When running the Podman virtual machine with rootful privileges (podman machine set --rootful=true), the errors mentioned above do not occur. However, Testcontainers for .NET is either unable to establish a connection to the Resource Reaper or fails:

  Docker API responded with status code=NotFound, response={"cause":"no such container","message":"no container with name or ID \"b26ef6bf452c4d85051d7727a2f53b2dddcf0c55b824aa03a6dd20b203ebc6f1\" found: no such container","response":404}

  • ✅ To fix the issue, you need to run the Podman virtual machine with root privileges and set the environment variables TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock and TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true.
• [ ] K3sContainer does not start (rancher/k3s:v1.26.2-k3s1)

  Failed to load kernel module br_netfilter with modprobe
  Failed to load kernel module iptable_nat with modprobe
  Failed to load kernel module iptable_filter with modprobe
  Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
  Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
  Failed to ApplyOOMScoreAdj" err="write /proc/self/oom_score_adj: permission denied
  Failed to set rlimit on max file handles" err="operation not permitted
  Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
  Failed to start ContainerManager" err="[open /proc/sys/vm/overcommit_memory: permission denied, open /proc/sys/kernel/panic: permission denied, open /proc/sys/kernel/panic_on_oops: permission denied]
  [...]

• [ ] LocalStackContainer does not start (localstack/localstack:1.4, localstack/localstack:2.0)

  2023-05-17T14:17:05.033  WARN --- [   asgi_gw_0] localstack.utils.archives  : Attempt 1. Failed to download archive from https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.zip: MyHTTPSConnectionPool(host='s3-us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /dynamodb-local/dynamodb_local_latest.zip (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff65444bb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
  2023-05-17T14:17:05.037  INFO --- [   asgi_gw_0] localstack.utils.archives  : Unable to extract file, re-downloading ZIP archive /tmp/localstack.ddb.zip: ('Failed to download archive from %s: . Retries exhausted', 'https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.zip')
  2023-05-17T14:17:25.079  WARN --- [   asgi_gw_0] localstack.utils.archives  : Attempt 1. Failed to download archive from https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.zip: MyHTTPSConnectionPool(host='s3-us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /dynamodb-local/dynamodb_local_latest.zip (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff65444d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
  2023-05-17T14:17:25.080  WARN --- [   asgi_gw_0] localstack.utils.functions : error calling function on_before_start: Installation of dynamodb-local failed.
  [...]

• [ ] Failed test:

    Failed DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.ContainerCleanUpStartsDefaultResourceReaper(resourceReaperEnabled: True) [1 ms]
    Error Message:
    Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"no such container","message":"container 02bb9d49a7b7ecd0501c4e1035c544c5ba77cd90b5596415a06be7dfc0cdf50a does not exist in database: no such container","response":500}
  
    Stack Trace:
      at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 447
    at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 217
    at Docker.DotNet.ContainerOperations.InspectContainerAsync(String id, CancellationToken cancellationToken) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 76
    at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStopAsync(CancellationToken ct) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 442
    at DotNet.Testcontainers.Containers.DockerContainer.DisposeAsyncCore() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 311
    at DotNet.Testcontainers.Resource.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Resource.cs:line 27
    at DotNet.Testcontainers.Containers.ResourceReaper.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/ResourceReaper.cs:line 170
    at DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.InitializeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.ResourceReaper.Tests/DefaultResourceReaperTest.cs:line 17

• [ ] Failed test:

    Failed DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.ContainerCleanUpStartsDefaultResourceReaper(resourceReaperEnabled: False) [1 ms]
    Error Message:
    Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"no such container","message":"container deca45c9a28733786162a3777bf145d0d00d37f335f7597d115a02186d5dd2a2 does not exist in database: no such container","response":500}
  
    Stack Trace:
      at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 447
    at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 217
    at Docker.DotNet.ContainerOperations.InspectContainerAsync(String id, CancellationToken cancellationToken) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 76
    at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStopAsync(CancellationToken ct) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 442
    at DotNet.Testcontainers.Containers.DockerContainer.DisposeAsyncCore() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 311
    at DotNet.Testcontainers.Resource.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Resource.cs:line 27
    at DotNet.Testcontainers.Containers.ResourceReaper.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/ResourceReaper.cs:line 170
    at DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.InitializeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.ResourceReaper.Tests/DefaultResourceReaperTest.cs:line 17

• [x] The test runs either with Docker or Podman if the mtu instead of the com.docker.network.driver.mtu key is used. Is the mtu a valid key for Docker (it is probably not that important since we only test if TC sets the property)?

    Failed DotNet.Testcontainers.Tests.Unit.TestcontainerNetworkBuilderTest.CreateNetworkAssignsOptions [38 ms]
    Error Message:
    Assert.Contains() Failure
  Not found: com.docker.network.driver.mtu
  In value:  KeyCollection<String, String> ["mtu"]
    Stack Trace:
      at DotNet.Testcontainers.Tests.Unit.TestcontainerNetworkBuilderTest.CreateNetworkAssignsOptions() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Networks/TestcontainerNetworkBuilderTest.cs:line 59

• [x] Cannot ping containers by its hostname (--hostname, WithHostname(string)), the ping commands fails with ping: bad address '_container2'. https://github.com/testcontainers/testcontainers-dotnet/blob/f0fef3ee3346665842426f1fe4bdead322c0097e/tests/Testcontainers.Tests/Unit/Networks/TestcontainersNetworkTest.cs#L47

    Failed DotNet.Testcontainers.Tests.Unit.TestcontainersNetworkTest.PingContainer(destination: "_testcontainer2") [85 ms]
    Error Message:
    Assert.Equal() Failure
  Expected: 0
  Actual:   1
    Stack Trace:
      at DotNet.Testcontainers.Tests.Unit.TestcontainersNetworkTest.PingContainer(String destination) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Networks/TestcontainersNetworkTest.cs:line 52

• [x] Failed test:

  • This looks like a race condition after the tests are finished (disposing the container).

    Failed DotNet.Testcontainers.Tests.Unit.TestcontainersContainerCancellationTest+Cancel.Start [1 ms]
    Error Message:
    [Test Class Cleanup Failure (DotNet.Testcontainers.Tests.Unit.TestcontainersContainerCancellationTest+Cancel)]: Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"no such container","message":"container 4c86760f485d02cb2fe7d1a20527fdc7c81fe6a9d24be63a3b16ea4884085310 does not exist in database: no such container","response":500}
    
    Stack Trace:
    at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 447
    at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 217
    at Docker.DotNet.ContainerOperations.InspectContainerAsync(String id, CancellationToken cancellationToken) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 76
    at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStopAsync(CancellationToken ct) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 442
    at DotNet.Testcontainers.Containers.DockerContainer.DisposeAsyncCore() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 311
    at DotNet.Testcontainers.Resource.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Resource.cs:line 27

• [ ] Podman can only set the mac address if the container is assigned to a network. .DependsOn(new NetworkBuilder().Build()) works. The Podman "workaround" does not work with Docker.

  This option can only be used if the <<container|pod>> is joined to only a single network

  Failed DotNet.Testcontainers.Tests.Unit.TestcontainersContainerTest+WithConfiguration.MacAddress [899 ms]
  Error Message:
  Assert.Equal() Failure
  ↓ (pos 0)
  Expected: 92:95:5e:30:fe:6d
  Actual:   ea:2c:b1:6c:16:c5
  ↑ (pos 0)
  Stack Trace:
  at DotNet.Testcontainers.Tests.Unit.TestcontainersContainerTest.WithConfiguration.MacAddress() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Containers/Unix/TestcontainersContainerTest.cs:line 107
  at DotNet.Testcontainers.Tests.Unit.TestcontainersContainerTest.WithConfiguration.MacAddress() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Containers/Unix/TestcontainersContainerTest.cs:line 108

Solution

-

Benefit

-

Alternatives

-

Would you like to help contributing this enhancement?

Yes

[tebeco]

Context: Apple MacBook Pro M3 - ARM Podman installed

might be a pre-requisite:

brew update
brew upgrade

# this is because QEMU had a bug since mid december / january that pulled an x86 image on ARM
# this is likely fixed and force a full re-creation
podman machine stop
podman machine rm
podman machine init

podman machine start

• did not touch any env var
• did not had to create any ssh tunnel

All I had to do was:

$ echo "ryuk.container.privileged = true" >> $HOME/.testcontainers.properties
$ cat $HOME/.testcontainers.properties

ryuk.container.privileged = true
$

and from podman Dashboard enable Docker compatibility:

[kanpov]

I've tried using Testcontainers with Podman via a simple ln /run/user/1000/podman/podman.sock /run/user/1000/docker.sock and so far it works exactly as well as with Docker, so are the issues only related to tests or are there any functional hindrances?

[HofmeisterAn]

so are the issues only related to tests or are there any functional hindrances?

The issues are not just related to the tests in this repository. Depending on your Podman environment and configuration, they can happen their as well.

[tebeco]

should this still be opened @HofmeisterAn ?

since qemu fixed their image

[HofmeisterAn]

I'm not sure, but in my opinion, we should keep the issue open until we have a working GitHub workflow that supports Podman. I haven't recently tested TC for .NET with Podman, I don't know which features are working and which are not.

[tebeco]

i fail to understand the relation between "testcontainer works with podman"✅ versus "adding a github workflow integration feature (to testcontainer withpod man")⚒️

that second one seems like an integration evol one that needs to be created as "proposal" while the second one seems working (this ticket)

[HofmeisterAn]

I think there are still a few issues that prevent Testcontainers for .NET from working properly with Podman, or at least obstacles. Ideally, this issue addresses the Podman incompatibilities (failing tests) and establishes a proper GH workflow (incl. updating the docs and explain which steps are necessary to set it up). Otherwise, we cannot know which features are working, which are not, and which may break with newer versions.

[tebeco]

/remindme 1year

🙃 (bad joke ok ...)

[mdx0111]

To get it working on Fedora 40, I ran the following command echo "ryuk.container.privileged = true" >> $HOME/.testcontainers.properties

Then ran this ln /run/user/1000/podman/podman.sock /run/user/1000/docker.sock

And everything started working nicely.

[serezlan]

Hi,

I'm facing problem when I create network with TC.

Steps too reproduce:

• create INetwork

Result: Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"CNI network \"1ddde97e-1272-4bf4-82c3-4998f4ab8c08\" not found","message":"error configuring network namespace for container aaeba79b34ab9de814ea4fa586696d507d72d47fd566c147748321f7a1285873: CNI network \"1ddde97e-1272-4bf4-82c3-4998f4ab8c08\" not found","response":500}

Environment: TC version : 3.10.0 dotnet 8 debian 12

I'm not sure if this is podman or TC issue. Please let me know if I need to make changes.

[serezlan]

apparently the default podman version in my distro is quite old (v3) I upgrade to v4.3.1 and I got it working.

thx :)