Bug: RYUK container startup failure when running Docker in rootless mode

[keneanung]

Describe the bug

When Docker is run in rootless mode, the ryuk fails to start as the docker socket mounted as a volume has the wrong permissions (nobody:nobody). This can be fixed be mounting the correct socket at /run/user/$(id -u)/docker.sock.

The overwrite can be done by setting the environment variable TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE to the above mentioned socket.

Ideally, this would be detected and configured automatically (ie. using Server.Security Options.rootless and Endpoints.docker.Host of the active context or the DOCKER_HOST environment variable).

If this is not an option, please consider documenting this case more in detail.

To Reproduce

Configure Docker to run in rootless mode (see https://docs.docker.com/engine/security/rootless/) and run the following snippet:

>>> from testcontainers.postgres import PostgresContainer
>>> with PostgresContainer() as container:
...   assert True

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

```bash # Get the operating system information (on a unix os). $ uname -a Linux FE-C-012RG 6.5.0-1019-oem #20-Ubuntu SMP PREEMPT_DYNAMIC Mon Mar 18 17:38:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux # Get the python version. $ python --version Python 3.12.2 # Get the docker version and other docker information. $ docker info Client: Docker Engine - Community Version: 26.0.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.13.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.26.1 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 19 Server Version: 26.0.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: false userxattr: true Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: e377cd56a71523140ca6ae87e30244719194a521 runc version: v1.1.12-0-g51d5e94 init version: de40ad0 Security Options: seccomp Profile: builtin rootless cgroupns Kernel Version: 6.5.0-1019-oem Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.32GiB Name: FE-C-012RG ID: 547c075a-39fa-4b4a-950a-b18069861839 Docker Root Dir: /home/CFL5FE/.local/share/docker Debug Mode: false Username: keneanung Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false # Get all python packages. $ pip freeze alembic==1.13.1 annotated-types==0.6.0 anyio==4.3.0 astroid==3.1.0 asyncpg==0.29.0 -e certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 coverage==7.4.4 cryptography==42.0.5 dill==0.3.8 docker==7.0.0 ecdsa==0.19.0 fastapi==0.110.1 fastapi-azure-auth==4.3.1 greenlet==3.0.3 h11==0.14.0 httpcore==1.0.5 httpx==0.27.0 hvac==2.1.0 idna==3.6 importlib_metadata==7.1.0 iniconfig==2.0.0 isort==5.13.2 Mako==1.3.2 MarkupSafe==2.1.5 mccabe==0.7.0 packaging==24.0 platformdirs==4.2.0 pluggy==1.4.0 pyasn1==0.6.0 pycparser==2.22 pydantic==2.6.4 pydantic_core==2.16.3 pylint==3.1.0 pytest==8.1.1 pytest-asyncio==0.23.6 pytest-cov==5.0.0 python-jose==3.3.0 PyYAML==6.0.1 requests==2.31.0 rsa==4.9 six==1.16.0 sniffio==1.3.1 SQLAlchemy==2.0.29 starlette==0.37.2 syrupy==4.6.1 testcontainers==4.3.3 tomli==2.0.1 tomlkit==0.12.4 typing_extensions==4.11.0 urllib3==2.2.1 uvicorn==0.29.0 vault-env-gen==0.2.0 wrapt==1.16.0 yapf==0.40.2 zipp==3.18.1 ```

[alexanderankin]

can you confirm this approach:

mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat >  detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection

def is_rootless(client: DockerClient):
  info = client.info()
  sec_opts = info.get('SecurityOptions') or tuple()

  return any('rootless' in s for s in sec_opts)

if __name__ == "__main__":
  print(is_rootless(from_env()))
EOF

python detect_rootless.py

[alexanderankin]

and then i guess we will need to tweak the Reaper class a bit in core

[keneanung]

can you confirm this approach:

mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat >  detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection

def is_rootless(client: DockerClient):
  info = client.info()
  sec_opts = info.get('SecurityOptions') or tuple()

  return any('rootless' in s for s in sec_opts)

if __name__ == "__main__":
  print(is_rootless(from_env()))
EOF

python detect_rootless.py

This returns the following:

$ python detect_rootless.py 
True

[flonix8]

I also use rootless Docker and I can confirm this issue. I can also confirm that setting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock fixes the issue. Alternatively, I have tried disabling Ryuk through TESTCONTAINERS_RYUK_DISABLED=true, which makes the tests run flawlessly as well.

[vemonet]

Would it be possible to fix this so that testcontainers just works with rootless docker? @alexanderankin

TESTCONTAINERS_RYUK_DISABLED=true pytest works, but it's a pain to use, and we can't even just os.environ['TESTCONTAINERS_RYUK_DISABLED'] = 'true' inside the python test file

Nowadays any dev slightly conscious should be using rootless docker (because it is safer, and because it reduce friction with file ownerships when mounting volumes inside an image that use the root user. So not only it is safer, but it is also easier to use...). It's a bit the future of containers, but it's already there and working. So testcontainers should be available for rootless docker without the need for complex setup

[alexanderankin]

you can import the config module and set it on the dataclass there

https://github.com/testcontainers/testcontainers-python/blob/85d6078f9bcc99050c0173e459208402aa4f5026/core/testcontainers/core/config.py#L51

• I'm not in a position where I'm forced to use rootless docker so I dont have easy access to an environment where I could easily test. if there is a guide or something that I can set up, this would be good prereq for contributions imo

[alexanderankin]

Ah, we have confirmed that the logic i provided above works. Then all that remains is plugging it in I suppose. I'll accept PR that fixes the issue. yes.

[vemonet]

I am not forced to use docker rootless neither, it's just that it's better than the root option, especially when you use containers a lot for development, try it you'll see :) Rootless docker can be easily enabled following these docs: https://docs.docker.com/engine/security/rootless/ usually 3 commands:

dockerd-rootless-setuptool.sh install
systemctl --user enable docker
loginctl enable-linger $UID

Unfortunately I just realized that ryuk is needed to stop the containers after running the tests. So disabling it is not a sustainable solution

But TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock works fine

An easy solution would be to just change the default RYUK_DOCKER_SOCKET depending on if rootless is detected or not here: https://github.com/testcontainers/testcontainers-python/blob/main/core/testcontainers/core/config.py#L15

I might look into this if I find sometimes and will send a PR