SELinux blocks bind mount with Docker Compose

[djnotes]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I have a service called db defined as follows in docker-compose.yml:

        db:
                image: mariadb:10.5
                environment:
                        MYSQL_ROOT_PASSWORD:  pass
                        MYSQL_DATABASE: user
                        MYSQL_USER: user
                        MYSQL_PASSWORD: pass
                networks:
                        - myapp_net
                ports: 
                        - 6603:3306
                volumes:
                        - ./data/:/docker-entrypoint-initdb.d:Z
                        - myapp_dbvol:/var/lib/mysql:Z
                        - ./data/conf.d:/etc/mysql/conf.d:Z

Steps to reproduce the issue:

1. Start podman.socket (systemctl --user start podman.socket)

2. Start docker-compose in the directory where docker-compose.yml exists

Describe the results you received: When starting docker-compose, backed by Podman, I get the following error:

db_1     |      mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 "Permission denied")

Describe the results you expected:

/etc/mysql/conf.d is inside the container, and not probably on the host. So I don't understand why SELinux does not like the container's access to it. Currently, I have to setenforce 0 to work around this, but wonder if there is a better solution.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

3.2.0

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-1.el8.3.4.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: codezombie
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 5.12.10-300.fc34.x86_64
  linkmode: dynamic
  memFree: 4007854080
  memTotal: 16671232000
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 16965951488
  swapTotal: 16965951488
  uptime: 31m 30.69s
registries:
  127.0.0.1:
    Blocked: false
    Insecure: true
    Location: 127.0.0.1
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: 127.0.0.1
  172.30.0.0/16:
    Blocked: false
    Insecure: true
    Location: 172.30.0.0/16
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: 172.30.0.0/16
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  configFile: /home/mehdi/.config/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 1
    stopped: 10
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/mehdi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 49
  runRoot: /run/user/1002/containers
  volumePath: /home/mehdi/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.0
  Built: 1623248656
  BuiltTime: Wed Jun  9 10:24:16 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.0-5.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

What AVCs are you seeing? Where does /etc/mysql/conf.d/ come from? Most likely this directory needs to have its labeling changed on the host.

[djnotes]

I'm mounting some custom configuration files from the host to /etc/mysql/conf.d for the mariadb container. About avc, I'm not sure. I just see the following error before the container exits:

db_1 | mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 "Permission denied")

[rhatdan]

Right the source content has to have the correct label on it. You can add a :Z to the volume in the compose, which should fix it. Same issue would happen with Docker Daemon if SELinux enabled.

[djnotes]

I'm using the :Z option in the docker-compose.yml file. However, looks like the SELinux option is not supported with services. Are there any workarounds?

[rhatdan]

That Error has nothing to do with Podman, I would think. Could you show me the line in the docker-compose.yml, we might have a bug in our code.

Anyways if you go to the directory on the host and do chcon -R -t container_file_t PATHTOHOSTDIR

It should solve your issue.

[djnotes]

The line causing error is:
- ./data/conf.d:/etc/mysql/conf.d:Z It seems the issue persists after running chcon.

[user]$ chcon -Rv -t container_file_t data
changing security context of 'data/conf.d/mariadb-extra.ini'
changing security context of 'data/conf.d'
changing security context of 'data/schema.sql'
changing security context of 'data'
[user]$ docker-compose up
Starting myapp_db_1    ... done
Starting myapp_redis_1 ... done
Starting myapp_app_1   ... done
Attaching to myapp_db_1, myapp_redis_1, myapp_app_1
db_1     | 2021-06-26 04:44:14+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
db_1     | 2021-06-26 04:44:14+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1     | 2021-06-26 04:44:14+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
db_1     | 2021-06-26 04:44:14+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
db_1     |      command was: mysqld --verbose --help --log-bin-index=/tmp/tmp.oQfFD6PXUf
db_1     |      mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 "Permission denied")
db_1     | Fatal error in defaults handling. Program aborted
redis_1  | 1:C 26 Jun 2021 04:44:14.318 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1  | 1:C 26 Jun 2021 04:44:14.318 # Redis version=6.2.1, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1  | 1:C 26 Jun 2021 04:44:14.318 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1  | 1:M 26 Jun 2021 04:44:14.319 * monotonic clock: POSIX clock_gettime
redis_1  | 1:M 26 Jun 2021 04:44:14.320 * Running mode=standalone, port=6379.
redis_1  | 1:M 26 Jun 2021 04:44:14.320 # Server initialized
redis_1  | 1:M 26 Jun 2021 04:44:14.321 * Loading RDB produced by version 6.2.1
redis_1  | 1:M 26 Jun 2021 04:44:14.321 * RDB age 148 seconds
redis_1  | 1:M 26 Jun 2021 04:44:14.321 * RDB memory usage when created 0.77 Mb
redis_1  | 1:M 26 Jun 2021 04:44:14.321 * DB loaded from disk: 0.000 seconds
redis_1  | 1:M 26 Jun 2021 04:44:14.321 * Ready to accept connections
myapp_db_1 exited with code 1

[rhatdan]

I believe this error is happening because the directory is now owned by the mariadb user within the container. If you went in the container the directory would be owned by root, and mariadb is running with the mariadb user.

If you add a :ZU to the volume mount, does that work?

[rhatdan]

./data/conf.d:/etc/mysql/conf.d:ZU

[djnotes]

No, still does not work. The error I get now when running docker-compose:

docker-compose up
Creating network "myapp_myapp_net" with the default driver
Creating myapp_db_1    ... error
Creating myapp_redis_1 ... 

ERROR: for myapp_db_1  Cannot create container for service db: fill out specgen: invaliCreating myapp_redis_1 ... done

ERROR: for db  Cannot create container for service db: fill out specgen: invalid option type "ZU"
ERROR: Encountered errors while bringing up the project.
[mehdi@codezombie myapp]$ docker-compose up
myapp_redis_1 is up-to-date
Creating myapp_db_1 ... error

ERROR: for myapp_db_1  Cannot create container for service db: fill out specgen: invalid option type "ZU"

ERROR: for db  Cannot create container for service db: fill out specgen: invalid option type "ZU"
ERROR: Encountered errors while bringing up the project.

[rhatdan]

Sorry they are supposed to be a comma separated list. Z,U

[djnotes]

Thank you for the clarification. You are right. Adding the Z,U options fixes the issue and docker-compose successfully launches the project.

[djnotes]

It's very weird. I just booted up my system and brought up the same project with Docker Compose, and the error shows up again. Adding more details from the error:

db_1     | 2021-06-30 04:49:37+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
db_1     | 2021-06-30 04:49:37+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1     | 2021-06-30 04:49:37+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
db_1     | 2021-06-30 04:49:37+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
db_1     |      command was: mysqld --verbose --help --log-bin-index=/tmp/tmp.Gt8rgYrQT2
db_1     |      mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 "Permission denied")
db_1     | Fatal error in defaults handling. Program aborted
tradebot_db_1 exited with code 1

I have made no changes in the db service of docker-compose.yml. The line is still there:
- ./data/conf.d:/etc/mysql/conf.d:Z,U Even tried docker-compose down and docker-compose up again. No difference.

[djnotes]

It's very weird. I just booted up my system and brought up the same project with Docker Compose, and the error shows up again. Adding more details from the error:

db_1     | 2021-06-30 04:49:37+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
db_1     | 2021-06-30 04:49:37+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1     | 2021-06-30 04:49:37+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
db_1     | 2021-06-30 04:49:37+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
db_1     |      command was: mysqld --verbose --help --log-bin-index=/tmp/tmp.Gt8rgYrQT2
db_1     |      mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 "Permission denied")
db_1     | Fatal error in defaults handling. Program aborted
tradebot_db_1 exited with code 1

I have made no changes in the db service of docker-compose.yml. The line is still there:
- ./data/conf.d:/etc/mysql/conf.d:Z,U Even tried docker-compose down and docker-compose up again. No difference.

After some time, without doing setenforce 0, after a few docker-compose down and docker-compose ups, it finally started successfully. Strange, isn't it?

[rhatdan]

If you are going to run multiple containers with the sam data directory then use :z so it can be shared amongst the containers.

[djnotes]

Strange. I'm still getting SELINUX errors, even with the z option.

SELinux is preventing iptables from ioctl access on the directory /sys/fs/cgroup.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that iptables should be allowed ioctl access on the cgroup directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'iptables' --raw | audit2allow -M my-iptables
# semodule -X 300 -i my-iptables.pp

Additional Information:
Source Context                unconfined_u:system_r:iptables_t:s0-s0:c0.c1023
Target Context                system_u:object_r:cgroup_t:s0
Target Objects                /sys/fs/cgroup [ dir ]
Source                        iptables
Source Path                   iptables
Port                          <Unknown>
Host                          mypc
Source RPM Packages           
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-34.11-1.fc34.noarch
Local Policy RPM              selinux-policy-targeted-34.11-1.fc34.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     mypc
Platform                      Linux mypc 5.12.12-300.fc34.x86_64 #1 SMP
                              Fri Jun 18 14:30:51 UTC 2021 x86_64 x86_64
Alert Count                   20426
First Seen                    2021-06-23 05:30:57 EDT
Last Seen                     2021-07-04 04:41:23 EDT
Local ID                      5e971c82-194b-428d-b60a-13e2cc94e346

Raw Audit Messages
type=AVC msg=audit(1625388083.328:7105): avc:  denied  { ioctl } for  pid=51655 comm="iptables" path="/sys/fs/cgroup" dev="cgroup2" ino=1 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 tcontext=system_u:object_r:cgroup_t:s0 tclass=dir permissive=0

Hash: iptables,iptables_t,cgroup_t,dir,ioctl

[rhatdan]

This AVC message is not related to the volume. Probably can be ignored. This looks like the iptables command was executed while sitting in a cgroup directory.

[pidement]

On my case, I solved a similar problem with host_volume/container_volume:z with z lowercase