Rootless Podman - SELinux Relabelling Issue

[zackman0010]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Our current setup uses the ~/.config/containers/storage.conf file to overwrite graphroot and rootless_storage_path to a folder /usr/local/csapps/podman/storage. This folder is owned by the csapp user. We do not have root access, so we are utilizing rootless Podman. Attempting to create a volume using podman volume create test fails with an error: error setting selinux label for /usr/local/csapps/podman/storage/volumes/test/_data to "system_u:object_r:container_file_t:s0:c171,c569" as shared: relabeling content in /usr is not allowed

We use the folder /usr/local/csapps as the central storage for all our applications, and that is where our system admins give us storage space to utilize. The only "workaround" we've found so far is to disable SELinux, but that's not feasible and our system admins would strongly prefer not to go that route. Is there anything we can do to allow us to use our existing folder structure without having to disable SELinux?

Steps to reproduce the issue:

1. Overwrite graphroot and rootless_storage_path to a folder nested under /usr

2. podman volume create test

Describe the results you received: error setting selinux label for /usr/local/csapps/podman/storage/volumes/test/_data to "system_u:object_r:container_file_t:s0:c171,c569" as shared: relabeling content in /usr is not allowed

Describe the results you expected: Volume to successfully be created

Additional information you deem important (e.g. issue happens only occasionally): I've recreated this error on a local device I do have root access to, so I can do any testing necessary.

Output of podman version:

Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.17.7
Built:        Wed Oct 12 09:42:59 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.26.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.3-1.module+el8.6.0+16771+28dfca77.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: 3571057ed895294b72709767db30b16722c8c24a'
  cpuUtilization:
    idlePercent: 99.13
    systemPercent: 0.33
    userPercent: 0.55
  cpus: 1
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-372.32.1.el8_6.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 107769856
  memTotal: 494022656
  networkBackend: netavark
  ociRuntime:
    name: runc
    package: runc-1.1.3-3.module+el8.6.0+16986+c8760fe3.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.3
      spec: 1.0.2-dev
      go: go1.17.12
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.6.0+16771+28dfca77.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 2602180608
  swapTotal: 2684350464
  uptime: 1h 21m 15.19s (Approximately 0.04 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/csapp/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /usr/local/csapps/podman/storage
  graphRootAllocated: 22441148416
  graphRootUsed: 2673348608
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/storage
  volumePath: /usr/local/csapps/podman/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1665582179
  BuiltTime: Wed Oct 12 09:42:59 2022
  GitCommit: ""
  GoVersion: go1.17.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

podman-4.1.1-7.module+el8.6.0+16889+9f154fa9.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

I've tested with the latest version available on the RHEL 8 repositories. Yes, I checked the Podman Troubleshooting Guide

[mheon]

@rhatdan PTAL

[rhatdan]

Does this help? https://github.com/containers/podman/blob/main/troubleshooting.md#11-changing-the-location-of-the-graphroot-leads-to-permission-denied

[zackman0010]

@rhatdan Unfortunately not, still same error after running those commands. The folders were relabeled successfully, but volume still would not create.

[zackman0010]

So I was looking through the code in the 4.1.1 tag and traced the error. The specific error being thrown is located in LabelVolumePath. The call that causes the error is SELinux's Relabel function, which ends up going down through the chcon function with recurse set to true. This ends up going to a badPrefix function, which explicitly excludes the /usr directory from being used as a prefix.

However, the logic in these functions has changed since 4.1.1, and that badPrefix function doesn't even exist anymore. I actually found the relevant commit in the SELinux repository.

Updating to the latest version of Podman would probably fix my issue, but unfortunately it's not available through the RHEL repositories yet. Is there anywhere I can download an RPM of the latest version, or do I have to build it from source?

[rhatdan]

4.2 should be available for RHEL very soon if not already, otherwise you need to build from source. Since this is fixed in the upstream I am closing.

[zackman0010]

Looks like the updated version of SELinux that allows /usr to be used isn't included until Podman 4.3, so I guess I'll see if our system admins will build it from the source. Thanks!

[rhatdan]

@mheon Any chance we can get this updated in podman 4.3.1?

[zackman0010]

Confirmed that it works after building version 4.3 from source. Thanks again for the help, sorry for the pointless issue!

[mheon]

@rhatdan Was this fixed in a specific PR? Is it just a new selinux vendor?

@ashley-cui FYI

[zackman0010]

The issue was fixed with the update from SELinux 1.10.1->1.10.2, which happened in this commit: https://github.com/containers/podman/commit/f00ceaabd497e87415f6926de50478260cd7e844

The SELinux commit that fixed it was this: https://github.com/opencontainers/selinux/commit/d3ccb697e01d1671de2f1dd32779efe8808e2132