Podman support

[gastaldi]

It would be really nice if TestContainers supported podman. This is a real game changer to no longer depend on Docker daemons to run.

[bsideup]

Thanks for asking! Since this is not the first time somebody asks for Podman, I would like to post a (non complete, most probably) list of requirements we need to do the Testcontainers' magic:

• [ ] Cross-platform support. It should work exactly the same on Linux, Mac and Windows, without having us do the VM management
• [ ] Docker networks, ability to start/connect/disconnect containers with/to Networks
• [ ] Image management - pulling (including the various authorisation mechanisms), listing, building
• [ ] Long running processes that survive the test run
• [ ] Access to /var/run/docker.sock from inside the container, ability to do regular Docker commands with it
• [ ] Log access/streaming
• [ ] Stdin/Stdout/Stderr streaming
• [ ] There are many Docker images that require root user, these should work too
• [ ] Volumes management

I am not a Podman expert, and if somebody from the community can help clarifying these, I would be happy to learn, and, as long as every point is clarified (the list is not 100% complete and we may need to add more in future) we can consider adding the support for Podman.

[bsideup]

and BTW, "daemonless" here makes it harder for Testcontainers :)

[gastaldi]

From my Podman experience:

• Access to /var/run/docker.sock : I don't think so, since there is no daemon;
• Cross Platform support: Podman only works in Linux. I haven't checked the internals of Testcontainers yet, but Podman offers an alias to replace the docker command, so Testcontainers could use Podman in Linux with the same CLI syntax as used in Docker (if the CLI is used);
• Image Building: is performed by Buildah

I wish @rhatdan could chime in the discussion to clarify or correct my statements :smiley:

[bsideup]

if the CLI is used

We're not using the CLI, but the REST API.

It looks to me that we should rather focus on waiting for the rootless Docker (an ongoing effort upstream), Podman as it is (plus Buildah, plus whatever else is needed) cannot provide the same functionality we need to do what we're doing in Testcontainers.

[Kidlike]

Dropping this here because I found it useful and relevant to the discussion :)

https://medium.com/nttlabs/cgroup-v2-596d035be4d7

[ppalaga]

We're not using the CLI, but the REST API.

Podman recently started offering a REST API: https://podman.io/blogs/2020/01/17/podman-new-api.html

[bsideup]

@ppalaga nice!

Do you have a compatibility table or something, to see which Docker's API endpoints are supported or not? Thanks!

[crunchtime-ali]

@bsideup I would help putting that table together. Is there a list of Docker API endpoints (and options) testcontainers uses or a test-suite we can run against podman's Docker compability API.

[bsideup]

@crunchtime-ali I believe running Testcontainers' test suite is the best way to check whether all APIs are supported or not. Our tests should cover all types of Docker features we are using.

You can also start a TCP proxy between Testcontainers and Podman to collect the list of endpoints that are used.

[rocketraman]

Some more information that might be helpful:

https://github.com/containers/libpod/blob/master/docs/source/markdown/podman-system-service.1.md

https://github.com/containers/libpod/blob/master/API.md

The Podman API supposedly has a docker API compatible endpoint.

[rocketraman]

And it can also expose a sock socket file as well, which should meet the requirement of having a /var/run/docker.sock (or equivalent) inside the container.

[rhatdan]

Correct we are under heavy development and are looking at testcontainers to prove that we handle the Docker API correctly. We have an extended version to support advanced features of Podman as well.

[cevich]

I'm also looking at doing the reverse: Running testcontainers tests in podman CI. Solving the challenges here, will help me get to there. (subscribed)

[baude]

how does one run the test suite? hints around how to specify the sock to connect to is a bonus. ill do the work.

[rocketraman]

@baude I'm not a testcontainers dev, but I suspect you can get insight into both of your questions from:

https://github.com/testcontainers/testcontainers-java/blob/master/.circleci/config.yml

and

https://github.com/testcontainers/testcontainers-java/blob/master/core/src/main/java/org/testcontainers/dockerclient/UnixSocketClientProviderStrategy.java

[bsideup]

@baude

Thanks for working on it!

we read the same environment variables as the Docker CLI does (as one of the strategies we support), e.g. DOCKER_HOST

By default, we also search for Docker environment in well-known locations (like /var/run/docker.sock)

Our primary CI is Azure Pipelines: https://github.com/testcontainers/testcontainers-java/blob/1.12.5/azure-pipelines.yml

[kiview]

@baude @rhatdan Great to see you get involved and that we can hope for great progress towards feature parity and Podman compatibility in Testcontainers. I know about a lot of Testcontainers users in more strict enterprise environments that would be very happy about Podman support 🙂

If there is any way I can support you, feel free to get in touch or join our Slack, if you want to have more extensive discussions.

[cevich]

Thanks @baude for taking this up, the java-stuff is out of my depth, but poke me on IRC if/when I can help with any automation-things.

[baude]

recent update here ... our compatibility layer requires the use of versioned paths (by api). the testcontainers suite does not appear to deal with that. i wrote a small java app using docker-java, which test containers also uses, and by default the versioned paths are not used there but it does provide the capability to do so with:

.withApiVersion("1.24")

So we need some sort of way to configure that in testcontainers and perhaps things will progress.

[baude]

update! a contributor has fixed ^^ so now just a matter of understanding how to dissect the test failures.

[ruddy32]

Defining DOCKER_HOST with /var/run/user/1000/podman/podman.sock generate this error: java.io.IOException: [111] Connection refused, even if the socket is created by the same Linux user account (running podman api with command systemctl --user start podman).

[bsideup]

Defining DOCKER_HOST with /var/run/user/1000/podman/podman.sock generate this error: java.io.IOException: [111] Connection refused, even if the socket is created by the same Linux user account (running podman api with command systemctl --user start podman).

Podman != Docker.

[Stexxen]

Just I'd mention that I came to this ticket after installing Fedora 32, realising I couldn't get docker-ce to run without lots of kernel/firewal changes https://github.com/docker/for-linux/issues/955

Looks like redhat are pushing podman https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/

So until/if the docker ticket is fixed, future fedora installations of docker are out of reach for layman users. However I'm wiping and rebuilding with 31, can't live without testcontainer ;-)

[bsideup]

Looks like redhat are pushing podman

It is so indeed, although Podman does not support all Docker's features and does not work with tools like Docker Compose, Fabric8 Docker Maven Plugin by Red Hat ( https://github.com/fabric8io/docker-maven-plugin/issues/1330 ), Testcontainers and others.

It is a bit sad to see a lot of push with statements like "just replace Docker CLI with Podman CLI and everything will be working" while, in reality, there is a ton of things missing. There were some attempts at adding Docker compatibility layer to Podman, but last information I heard was that it does not pass Testcontainers' test suite.

However I'm wiping and rebuilding with 31, can't live without testcontainer ;-)

Thank you! FYI @kiview is running Fedora 32 with Docker, he just had to apply a couple of modifications (cgroups v1, firewall settings)

---

Also FYI, I did some experiments with Rootless Docker and apparently it is working! (in PoC mode) I will explore further because I believe that this is the way to go - good old Docker, just without the root requirement.

[ppalaga]

It is a bit sad to see a lot of push with statements like "just replace Docker CLI with Podman CLI and everything will be working" while, in reality, there is a ton of things missing.

Could you please list which specific features Podman is missing to be able to serve as a replacement for Docker in Testcontainers?

[bsideup]

@ppalaga sorry, but I don't know (I am on Mac and can't use Podman to even test)

perhaps ask @baude: https://github.com/testcontainers/testcontainers-java/issues/2088#issuecomment-595960775

Also, If Podman serves as a drop-in replacement for Docker (as claimed), it should be trivial to test it - just run Testcontainers' and docker-java's test suites with it.

[rhatdan]

The original goal was Podman as a drop in replacement for Docker CLI. Which it has done a pretty damn good job at. The next level is to have Podman implement the Docker API, which is ongoing and heavily being worked on right now. We call this APIV2. You should be seeing Release candidates for Podman 2.0 right now, where we are beginning the testing on it. If people could test suites like Testcontainers and docker-java's test suite, we would like to see the results.

Potential issues we have seen so far are:

• Test suites like docker-py which rely on Docker-swarm, which we don't intend on implementing.
• Docker-compose has issues since Podman networking and Docker networking are very different. we are attempting to work around these issues, to see where we are on these test suites.

[rhatdan]

Bottom line on this, is we need help from the community to help with tests, and fixes to get to the point where we can support the Docker.sock API.

[bsideup]

@rhatdan Thanks for the update!

If someone from your side can help me configuring Podman on GitHub Actions, I could add a branch to docker-java and run the test suite (excluding the Swarm Mode tests) Once done, I can do the same for Testcontainers 👍

[rhatdan]

@cevich PTAL @bsideup Thanks.

[baude]

@bsideup what do you need? a github notification set up?

[bsideup]

@baude some example of running GitHub Actions with Podman configured

[ruddy32]

After restarting the linux system, I get this error:

May 26, 2020 11:44:25 PM org.testcontainers.dockerclient.DockerClientProviderStrategy lambda$getFirstValidStrategy$2
INFO: Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///run/user/1000/podman/podman.sock
...
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageNameFuture=java.util.concurrent.CompletableFuture@74ad8d05[Completed normally], imagePullPolicy=DefaultPullPolicy(), dockerClient=LazyDockerClient.INSTANCE)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1265)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:600)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:311)
    ... 45 more
Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: {"cause":"no such image","message":"NewFromLocal(): unable to find 'quay.io/testcontainers/ryuk:0.2.3' in local storage: no such image","response":500}

It sounds better but it might be a problem with podman config or something else.

I will do a new test with podman 1.9.3-2 (archlinux), and maybe 2.0

[rhatdan]

Is there a reason to use rootless podman? I would figure the equivalence is with rootful?

[ruddy32]

Using the system podman API socket fail with Caused by: java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied Even changing socket acl wth 0666.

[skorhone]

@ruddy32 Does API socket respond to curl? See https://liquidat.wordpress.com/2020/04/20/howto-using-the-new-podman-api/amp/

Your previous "no such image" error was probably due to misconfigured registries. Podman doesn't default to docker.io. I guess you figured that one already though

[ruddy32]

Using API with curl works fine.

Repository configuration is setup with registries = ['docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com', 'registry.centos.org'].

I undertand this configuration is ok for quay.io.

[skorhone]

Finally had some time to test this. There seems to be few issues with podman's current implementation:

• In podman created timestamp for image is a string ( 2020-06-23T11:06:02.096204727Z ) in docker it's an integer (long)
• Podman doesn't seem to support filters for images ( e.g. http://localhost/images/json?filter=testcontainersofficial%2Fryuk%3A0.3.0 )

I'll report these issues to containers/libpod.

[skorhone]

containers/libpod#6796 containers/libpod#6797

[skorhone]

docker-java/docker-java#1424

[skorhone]

Patched docker-java locally and now it succeeds to pull ryuk image, but crashes when it starts to launch it. There's something wrong with podman service, when creating rootless containers (could be os specific).

Issue documented here: containers/libpod#6798

[skorhone]

Unfortunately there's yet another issue: containers/libpod#6799

My next thought was to disable ryuk for now. However that didn't get me much further. There also seems to be an issue related to polling of image pull status. I don't bother to report that before container create endpoint works as expected.

Something is also broken in testcontainers startup checks :-) While checks be bypassed with a configuration option, it makes sense to fix them.

While there are multiple issues, I don't think that situation looks hopeless at all. RedHat is doing awesome job coordinating podman efforts. If I look statistics correctly, there's more activity on podman than there is on moby. And then there's buildah, fuse-overlayfs and crio too :-)

[skorhone]

Finally had time to set environment to work with a debugger. This speeds things up from my perspective.

Podman doesn't currently expose port information the same way that docker does: containers/libpod#6803

[ricardozanini]

Looks like podman has good news: https://podman.io/blogs/2020/07/01/rest-versioning.html

[skorhone]

@ricardozanini Rest API is the one that I'm testing against 🙂

There's still few blockers, most notable are issues with container creation and exec endpoints. I was told that first one will be resolved by major refactoring that unifies code paths for libpod (podman) and compat (docker) API handlers for container creation. The issue with exec API was just recently posted, so that might take a while to be resolved.

I also believe there might be compatibility issue with image pulling related to way that pull progress is being monitored by docker-java. I didn't file an issue for that yet, since other issues are of higher priority (documenting issue takes quite a bit of time)

I might be moving on to test GitLab runner while these issues are being resolved 🙂

[skorhone]

Exec issues were fixed with a simple patch. There is still some kind of issue with either connection management or connection tracking (podman connections counts show that there are active connections while netstat shows none)

Here's a first passing "test":

@Testcontainers
public class NotATest {
    public static class SomeContainer extends GenericContainer<SomeContainer> {
        public SomeContainer() {
            super("redis:5.0.3-alpine");
            withExposedPorts(6379);
            setPortBindings(Collections.singletonList("6379:6379/tcp"));
            setCommand("docker-entrypoint.sh", "redis-server");
        }
    }

    // container {
    @Container
    public GenericContainer<?> redis = new SomeContainer();

    @BeforeEach
    public void setUp() {
        ch.qos.logback.classic.Logger tc = (ch.qos.logback.classic.Logger) org.slf4j.LoggerFactory.getLogger("org.testcontainers");
        tc.setLevel(Level.ALL);
        ch.qos.logback.classic.Logger dj = (ch.qos.logback.classic.Logger) org.slf4j.LoggerFactory.getLogger("com.github.dockerjava");
        dj.setLevel(Level.ALL);
        ch.qos.logback.classic.Logger ap = (ch.qos.logback.classic.Logger) org.slf4j.LoggerFactory.getLogger("org.apache.http");
        ap.setLevel(Level.ALL);

        String address = redis.getHost();
        Integer port = redis.getFirstMappedPort();

        System.out.println(address + ":" + port);
    }

    @Test
    public void testSimpleSetAndGet() {
        Jedis jedis = new Jedis("localhost");

        jedis.set("foo", "yes");
        System.out.println("Redis is a live: " + jedis.get("foo"));

        jedis.close();
    }
}

There is a pull request pending that is supposed to fix setCommand part, but for now it's mandatory to define command to prevent podman creating invalid container. Publish all flag wasn't working correctly last time I checked - which explains why I'm calling both expose and setPortBindings. There might be a fair bit of work to be done to fix publish all, considering that libpod and podman's v2 (docker) api use currently different code paths. Libpod api seems far more reliable than v2 at the moment.

For now I'm running with TESTCONTAINERS_RYUK_DISABLED set to true.

[skorhone]

containers/podman#6835 fixed requirement to use setCommand. So now you need to look for containers/podman#6918 which fixes some leaks related to use of exec. There's currently no issue open for exposing ports - and unfortunately I will be soon off for summer holidays (4weeks) I did ping person that fixed setCommand and he might be able to fix issue with ports.

[zhangguanzhang]

containers/podman#6835 fixed requirement to use setCommand. So now you need to look for containers/podman#6918 which fixes some leaks related to use of exec. There's currently no issue open for exposing ports - and unfortunately I will be soon off for summer holidays (4weeks) I did ping person that fixed setCommand and he might be able to fix issue with ports.

https://github.com/containers/podman/pull/6835#issuecomment-656194612 @skorhone Could you show the steps how to to reproduce the error?

[skorhone]

@zhangguanzhang If you run testcase I posted few comments back and remove setPortBindings, you should be able to replicate behavior that I see. Be sure to disable that ryuk

I'm running podman in my testcontainers tests just like I've done in: https://github.com/skorhone/libpod-gitlab-it/blob/cirrus/.cirrus.yml . TCP binding makes it easier to capture API call. docker-java supports reading target from DOCKER_HOST so it's trivial to use TCP with it.

If you want to build test case manually, you need to get a container that has a service that is binding to TCP port. e.g. netcat (nc). Then expose port and set publish all ports to true. Exposed port should be published to a randomized port

[zhangguanzhang]

@zhangguanzhang If you run testcase I posted few comments back and remove setPortBindings, you should be able to replicate behavior that I see. Be sure to disable that ryuk

I'm running podman in my testcontainers tests just like I've done in: https://github.com/skorhone/libpod-gitlab-it/blob/cirrus/.cirrus.yml . TCP binding makes it easier to capture API call. docker-java supports reading target from DOCKER_HOST so it's trivial to use TCP with it.

If you want to build test case manually, you need to get a container that has a service that is binding to TCP port. e.g. netcat (nc). Then expose port and set publish all ports to true. Exposed port should be published to a randomized port

I see this in ci:

panic: Local repo not found, please run `make development_setup`

could you retry test with run the make before the testing?