[Enhancement]: Extending Testcontainers to support multiple container runtimes (Podman)

HofmeisterAn commented 1 year ago

Problem

Since Testcontainers for Node has done an awesome job supporting multiple container runtimes (not just Docker), it makes perfect sense to continue this great work and look into implementing it for .NET. We can start by collecting a list of tasks, issues or incompabilities that are necessary to support other runtimes as well.

[x] Docker.DotNet's ExtractArchiveToContainerAsync(string, ContainerPathStatParameters, Stream, CancellationToken) throws System.Net.Sockets.SocketException : Connection reset by peer.
- ✅ https://github.com/dotnet/Docker.DotNet/pull/629
[x] Running Testcontainers for .NET on macOS using Podman (brew installation) does not work well with the Resource Reaper. Testcontainers is incapable of mounting the daemon socket. It fails with different errors:
```
Docker API responded with status code=InternalServerError, response={"cause":"operation not supported","message":"container create: statfs /Users/atomicjar/.local/share/containers/podman/machine/qemu/podman.sock: operation not supported","response":500}
```
When attempting to use the daemon socket path from within the Podman virtual machine, the process is also unsuccessful. Setting export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock fails:
```
Docker API responded with status code=InternalServerError, response={"cause":"permission denied","message":"container create: statfs /var/run/docker.sock: permission denied","response":500}
```
When running the Podman virtual machine with rootful privileges (podman machine set --rootful=true), the errors mentioned above do not occur. However, Testcontainers for .NET is either unable to establish a connection to the Resource Reaper or fails:
```
Docker API responded with status code=NotFound, response={"cause":"no such container","message":"no container with name or ID \"b26ef6bf452c4d85051d7727a2f53b2dddcf0c55b824aa03a6dd20b203ebc6f1\" found: no such container","response":404}
```
- ✅ To fix the issue, you need to run the Podman virtual machine with root privileges and set the environment variables TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock and TESTCONTAINERS_RYUK_CONTAINER_PRIVILEGED=true.

[ ] K3sContainer does not start (rancher/k3s:v1.26.2-k3s1)

Failed to load kernel module br_netfilter with modprobe
Failed to load kernel module iptable_nat with modprobe
Failed to load kernel module iptable_filter with modprobe
Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
Failed to ApplyOOMScoreAdj" err="write /proc/self/oom_score_adj: permission denied
Failed to set rlimit on max file handles" err="operation not permitted
Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
Failed to start ContainerManager" err="[open /proc/sys/vm/overcommit_memory: permission denied, open /proc/sys/kernel/panic: permission denied, open /proc/sys/kernel/panic_on_oops: permission denied]
[...]

[ ] LocalStackContainer does not start (localstack/localstack:1.4, localstack/localstack:2.0)

2023-05-17T14:17:05.033  WARN --- [   asgi_gw_0] localstack.utils.archives  : Attempt 1. Failed to download archive from https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.zip: MyHTTPSConnectionPool(host='s3-us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /dynamodb-local/dynamodb_local_latest.zip (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff65444bb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2023-05-17T14:17:05.037  INFO --- [   asgi_gw_0] localstack.utils.archives  : Unable to extract file, re-downloading ZIP archive /tmp/localstack.ddb.zip: ('Failed to download archive from %s: . Retries exhausted', 'https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.zip')
2023-05-17T14:17:25.079  WARN --- [   asgi_gw_0] localstack.utils.archives  : Attempt 1. Failed to download archive from https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.zip: MyHTTPSConnectionPool(host='s3-us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /dynamodb-local/dynamodb_local_latest.zip (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff65444d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2023-05-17T14:17:25.080  WARN --- [   asgi_gw_0] localstack.utils.functions : error calling function on_before_start: Installation of dynamodb-local failed.
[...]

[ ] Failed test:

  Failed DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.ContainerCleanUpStartsDefaultResourceReaper(resourceReaperEnabled: True) [1 ms]
  Error Message:
  Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"no such container","message":"container 02bb9d49a7b7ecd0501c4e1035c544c5ba77cd90b5596415a06be7dfc0cdf50a does not exist in database: no such container","response":500}

  Stack Trace:
    at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 447
  at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 217
  at Docker.DotNet.ContainerOperations.InspectContainerAsync(String id, CancellationToken cancellationToken) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 76
  at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStopAsync(CancellationToken ct) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 442
  at DotNet.Testcontainers.Containers.DockerContainer.DisposeAsyncCore() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 311
  at DotNet.Testcontainers.Resource.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Resource.cs:line 27
  at DotNet.Testcontainers.Containers.ResourceReaper.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/ResourceReaper.cs:line 170
  at DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.InitializeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.ResourceReaper.Tests/DefaultResourceReaperTest.cs:line 17

[ ] Failed test:

  Failed DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.ContainerCleanUpStartsDefaultResourceReaper(resourceReaperEnabled: False) [1 ms]
  Error Message:
  Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"no such container","message":"container deca45c9a28733786162a3777bf145d0d00d37f335f7597d115a02186d5dd2a2 does not exist in database: no such container","response":500}

  Stack Trace:
    at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 447
  at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 217
  at Docker.DotNet.ContainerOperations.InspectContainerAsync(String id, CancellationToken cancellationToken) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 76
  at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStopAsync(CancellationToken ct) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 442
  at DotNet.Testcontainers.Containers.DockerContainer.DisposeAsyncCore() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 311
  at DotNet.Testcontainers.Resource.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Resource.cs:line 27
  at DotNet.Testcontainers.Containers.ResourceReaper.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/ResourceReaper.cs:line 170
  at DotNet.Testcontainers.ResourceReaper.Tests.DefaultResourceReaperTest.InitializeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.ResourceReaper.Tests/DefaultResourceReaperTest.cs:line 17

[x] The test runs either with Docker or Podman if the mtu instead of the com.docker.network.driver.mtu key is used. Is the mtu a valid key for Docker (it is probably not that important since we only test if TC sets the property)?

  Failed DotNet.Testcontainers.Tests.Unit.TestcontainerNetworkBuilderTest.CreateNetworkAssignsOptions [38 ms]
  Error Message:
  Assert.Contains() Failure
Not found: com.docker.network.driver.mtu
In value:  KeyCollection<String, String> ["mtu"]
  Stack Trace:
    at DotNet.Testcontainers.Tests.Unit.TestcontainerNetworkBuilderTest.CreateNetworkAssignsOptions() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Networks/TestcontainerNetworkBuilderTest.cs:line 59

[x] Cannot ping containers by its hostname (--hostname, WithHostname(string)), the ping commands fails with ping: bad address '_container2'. https://github.com/testcontainers/testcontainers-dotnet/blob/f0fef3ee3346665842426f1fe4bdead322c0097e/tests/Testcontainers.Tests/Unit/Networks/TestcontainersNetworkTest.cs#L47

  Failed DotNet.Testcontainers.Tests.Unit.TestcontainersNetworkTest.PingContainer(destination: "_testcontainer2") [85 ms]
  Error Message:
  Assert.Equal() Failure
Expected: 0
Actual:   1
  Stack Trace:
    at DotNet.Testcontainers.Tests.Unit.TestcontainersNetworkTest.PingContainer(String destination) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Networks/TestcontainersNetworkTest.cs:line 52

[x] Failed test:

This looks like a race condition after the tests are finished (disposing the container).

Failed DotNet.Testcontainers.Tests.Unit.TestcontainersContainerCancellationTest+Cancel.Start [1 ms]
Error Message:
[Test Class Cleanup Failure (DotNet.Testcontainers.Tests.Unit.TestcontainersContainerCancellationTest+Cancel)]: Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"cause":"no such container","message":"container 4c86760f485d02cb2fe7d1a20527fdc7c81fe6a9d24be63a3b16ea4884085310 does not exist in database: no such container","response":500}

Stack Trace:
at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 447
at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/DockerClient.cs:line 217
at Docker.DotNet.ContainerOperations.InspectContainerAsync(String id, CancellationToken cancellationToken) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 76
at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStopAsync(CancellationToken ct) in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 442
at DotNet.Testcontainers.Containers.DockerContainer.DisposeAsyncCore() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 311
at DotNet.Testcontainers.Resource.DisposeAsync() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/src/Testcontainers/Resource.cs:line 27

[ ] Podman can only set the mac address if the container is assigned to a network. .DependsOn(new NetworkBuilder().Build()) works. The Podman "workaround" does not work with Docker.

This option can only be used if the <<container|pod>> is joined to only a single network

Failed DotNet.Testcontainers.Tests.Unit.TestcontainersContainerTest+WithConfiguration.MacAddress [899 ms]
Error Message:
Assert.Equal() Failure
↓ (pos 0)
Expected: 92:95:5e:30:fe:6d
Actual:   ea:2c:b1:6c:16:c5
↑ (pos 0)
Stack Trace:
at DotNet.Testcontainers.Tests.Unit.TestcontainersContainerTest.WithConfiguration.MacAddress() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Containers/Unix/TestcontainersContainerTest.cs:line 107
at DotNet.Testcontainers.Tests.Unit.TestcontainersContainerTest.WithConfiguration.MacAddress() in /home/runner/work/testcontainers-dotnet/testcontainers-dotnet/tests/Testcontainers.Tests/Unit/Containers/Unix/TestcontainersContainerTest.cs:line 108

Solution

-

Benefit

-

Alternatives

-

Would you like to help contributing this enhancement?

Yes

tebeco commented 7 months ago

Context: Apple MacBook Pro M3 - ARM Podman installed

might be a pre-requisite:

brew update
brew upgrade

# this is because QEMU had a bug since mid december / january that pulled an x86 image on ARM
# this is likely fixed and force a full re-creation
podman machine stop
podman machine rm
podman machine init

podman machine start

did not touch any env var
did not had to create any ssh tunnel

All I had to do was:

$ echo "ryuk.container.privileged = true" >> $HOME/.testcontainers.properties
$ cat $HOME/.testcontainers.properties

ryuk.container.privileged = true
$

and from podman Dashboard enable Docker compatibility:

kanpov commented 3 months ago

I've tried using Testcontainers with Podman via a simple ln /run/user/1000/podman/podman.sock /run/user/1000/docker.sock and so far it works exactly as well as with Docker, so are the issues only related to tests or are there any functional hindrances?

HofmeisterAn commented 3 months ago

so are the issues only related to tests or are there any functional hindrances?

The issues are not just related to the tests in this repository. Depending on your Podman environment and configuration, they can happen their as well.

tebeco commented 3 weeks ago

should this still be opened @HofmeisterAn ?

since qemu fixed their image

HofmeisterAn commented 3 weeks ago

I'm not sure, but in my opinion, we should keep the issue open until we have a working GitHub workflow that supports Podman. I haven't recently tested TC for .NET with Podman, I don't know which features are working and which are not.

tebeco commented 2 weeks ago

i fail to understand the relation between "testcontainer works with podman"✅ versus "adding a github workflow integration feature (to testcontainer withpod man")⚒️

that second one seems like an integration evol one that needs to be created as "proposal" while the second one seems working (this ticket)

HofmeisterAn commented 2 weeks ago

I think there are still a few issues that prevent Testcontainers for .NET from working properly with Podman, or at least obstacles. Ideally, this issue addresses the Podman incompatibilities (failing tests) and establishes a proper GH workflow (incl. updating the docs and explain which steps are necessary to set it up). Otherwise, we cannot know which features are working, which are not, and which may break with newer versions.

tebeco commented 2 weeks ago

/remindme 1year

🙃 (bad joke ok ...)

mdx0111 commented 5 hours ago

To get it working on Fedora 40, I ran the following command echo "ryuk.container.privileged = true" >> $HOME/.testcontainers.properties

Then ran this ln /run/user/1000/podman/podman.sock /run/user/1000/docker.sock

And everything started working nicely.

testcontainers / testcontainers-dotnet