Podman guide needs some discussion of Ryuk

[holly-cummins]

This is a complicated area, and easy to get wrong, so raising an issue to get it all written down.

Before Quarkus 2.13 (or therabouts), to successfully run a simple app like the todo app with Podman, you needed to have

ryuk.container.privileged=true

in your testcontainers.properties. To reproduce this, edit ~/.testcontainers.properties to make sure ryuk.container.privileged=true isn't in there, and then run:

git clone git@github.com:cescoffier/quarkus-todo-app.git
quarkus-todo-app
git checkout 9c690c766790f650a53bf872de474364ef9db14a
cd quarkus-todo
mvn install

The build will fail with an error about not being able to contact ryuk. If you add back ryuk.container.privileged=true and run mvn install, the build will pass and you'll see a ryuk container appear, and then disappear, in podman desktop.

The other way to make the build pass used to be to set TESTCONTAINERS_RYUK_DISABLED="true" as an environment variable, but the testcontainers team really prefer that people don't do that.

Since then, two things definitely changed:

• @geoand did a nice fix in response to #25852; he'd spotted that (a) even checking the docker environment was enough to trigger testcontainers to start ryuk, which is silly and (b) setting reuse to true 'temporarily' disables ryuk ... so we now make sure that reuse is set to true when checking docker
• the reuse configuration for datasource dev services changed, although I haven't traced down the PR or why we did it, so that we don't have quarkus.datasource.*.devservices.reuse configuration anymore

So it used to be Ryuk would always start, but now, in more recent Quarkus versions, we usually don't start Ryuk, because most of the dev services have reuse set to true, and we fixed the problem where just figuring out if we had docker caused Ryuk to start.

But there are still paths where we get Ryuk-related failures. If you delete ~/.testcontainers.properties entirely, and then run the todo app, the 'local' reuse settings can't take effect, because 'global' reuse is off:

rm ~/.testcontainers.properties
git clone git@github.com:cescoffier/quarkus-todo-app.git
cd quarkus-todo-app
cd quarkus-todo
mvn install

You can also achieve something similar by editing the settings to disable global re-use, testcontainers.reuse.enable=false, and then running mvn install, but that's a bit more artificial:

2024-08-13 11:44:14,110 WARN  [tc.doc.io/postgres:14] (build-36) Reuse was requested but the environment does not support the reuse of containers
To enable reuse of containers, you must set 'testcontainers.reuse.enable=true' in a file located at /Users/holly/.testcontainers.properties

The testcontainers properties file is hard to reason about in a sensible way, because it gets edited during a Quarkus run, and it's hard to work out which of the contents were changed because of Quarkus things, and which were changed by Testcontainers, or if any of them were sensible defaults. See https://github.com/quarkusio/quarkus/issues/34294 for an issue where Ryuk is causing problems and updating the testcontainers properties fixes it, but only temporarily, because something changes it back.

I also wondered if we'd have Ryuk-related problems with devservices that do not enable reuse by default, but there's not so many of those. Infinispan doesn't enable re-use, so I tried the infinispan codestart with Quarkus. Everything works, but I noticed that Infinispan comes with it's own Ryuk image (maybe with different privilege requirements?):

Given that:

• Removing ~/.testcontainers.properties causes Ryuk-related failures even for trivial apps
• We don't know what the behaviour is on a clean machine
• We don't know for sure that all of our dev services either set reuse to true or ship with their own Ryuk

... I think we have to reinstate some of the discussion of Ryuk back into our Podman guide that was removed by https://github.com/quarkusio/quarkus/pull/39692. Otherwise people will have to look elsewhere for debug information if they hit Ryuk failures, which will be frustrating for them. We could move it to a troubleshooting section, since people may not have to do it, but it would be nice to test on a clean machine to confirm that.

I think the consequences of running Ryuk privileged are low - the Testcontainers docs say it has to be run as a privileged container, but if there are arguments against it we should also get them written down in this issue (or another one).

[quarkus-bot[bot]]

/cc @maxandersen (podman)

[kiview]

Thanks for writing things down @holly-cummins, trying to prove some more context from the Testcontainers perspective.

I think the docs regarding Ryuk being a privileged container are a bit unclear from our side. The essential things that Ryuk needs, is mounted access to a Docker socket, for interacting with the Docker daemon. And there are different reasons why this might fail or not be allowed in some environments, sometimes because only a privileged container is allowed to mount the Docker socket.

I don't know what a privileged container means in the context of Podman.

From the Testcontainers side, we have an interest in providing Quarkus with a good integration point and, if anything, degrade in a graceful way for Podman users, instead of failing with hard to debug errors. Ideally we would like to have Quarkus avoid messing with ~/.testcontainers.properties, since it is also edited by Testcontainers itself, as well as by Testcontainers Desktop in certain cases.

verything works, but I noticed that Infinispan comes with it's own Ryuk image (maybe with different privilege requirements?):

TBH, that somehow looks like it is pulling in a very old tc-java version, with the corresponding old image?