search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.92k stars 2.42k forks source link

Podman with `fuse-overlayfs` and `force_mask = "0700"` mangles ownership and permissions inside running containers #22810

Open jiridanek opened 6 months ago

jiridanek commented 6 months ago

Issue Description

$ cat ~/.config/containers/containers.conf 
[containers]

[engine]

[machine]

[network]
default_rootless_network_cmd="slirp4netns"

[secrets]

[configmaps]
$ cat ~/.config/containers/storage.conf 
[storage]
driver="overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
force_mask = "0700"

Steps to reproduce the issue

Steps to reproduce the issue

  1. apply the configs from above
  2. run the command from below

Describe the results you received

$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/jdanek/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y
 A "/home/jdanek/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.
$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 9caf8ba21fce done   | 
Copying config 5858ca0e9f done   | 
Writing manifest to image destination
drwx------. 2 root root 104 May 26 08:29 /home/pepa/
drwx------. 2 pepa nobody 104 May 26 08:29 /home/pepa/
sudo: unable to execute /usr/bin/ls: Permission denied

Describe the results you expected

After I comment out the force_mask = "0700", or delete storage.conf alltogether, I get

$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/jdanek/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y
 A "/home/jdanek/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.
$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 9caf8ba21fce done   | 
Copying config 5858ca0e9f done   | 
Writing manifest to image destination
drwx------. 2 pepa pepa 104 May 26 08:32 /home/pepa/
drwx------. 2 pepa pepa 104 May 26 08:32 /home/pepa/
pepa

podman info output

$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 85.42
    systemPercent: 1.81
    userPercent: 12.76
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 6.8.9-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 5639622656
  memTotal: 33392324608
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 27000832
  swapTotal: 8589930496
  uptime: 145h 13m 23.00s (Approximately 6.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  image-registry.openshift-image-registry.svc:
    Blocked: false
    Insecure: false
    Location: default-route-openshift-image-registry.apps-crc.testing
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: image-registry.openshift-image-registry.svc
    PullFromMirror: ""
store:
  configFile: /home/jdanek/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: "0700"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc40.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
  graphRoot: /home/jdanek/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 400291479552
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/jdanek/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Fri May 10 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

I use btrfs on Fedora 40.

I discovered this when exploring around

In the issue I am reporting now, I deliberately did not use any networked filesystem but still got weird permissions/ownership errors.

Additional information

Article https://www.redhat.com/sysadmin/podman-rootless-overlay suggests that fuse-overlayfs is supposed to preserve permissions and ownership in file xattrs.

This issue seems to be similar to a several previous issues

And it might be related to

legobeat commented 5 months ago

Might this be the same issue (this is with force_mask = "shared", though)? Permissions inside build container get reset to root when switching user on rootless build:

$ sudo getenforce
Disabled

$ cat ~/.config/containers/storage.conf
[storage]
driver = "overlay"
[storage.options.overlay]
force_mask = "shared"
mount_program = "/usr/bin/fuse-overlayfs"

$ cat Containerfile.testperms
FROM alpine:3.19
RUN mkdir -p /foo && chmod 777 /foo && chown 1234:1234 /foo && ls -la /foo
USER 1234
RUN ls -la /foo && touch /foo/bar

$ podman build --no-cache -t testperms -f Containerfile.testperms
STEP 1/4: FROM alpine:3.19
STEP 2/4: RUN mkdir -p /foo && chmod 777 /foo && chown 1234:1234 /foo && ls -la /foo
total 12
drwxrwxrwx    2 1234     1234          4096 Jun 12 05:19 .
drwxr-xr-x   20 root     root          4096 Jun 12 05:19 ..
--> 8da48daca6ea
STEP 3/4: USER 1234
--> b6088de82f9c
STEP 4/4: RUN ls -la /foo && touch /foo/bar
total 8
drwxr-xr-x    2 root     root          4096 Jun 12 05:19 .
drwxr-xr-x   20 root     root          4096 Jun 12 05:19 ..
touch: /foo/bar: Permission denied
Error: building at STEP "RUN ls -la /foo && touch /foo/bar": while running runtime: exit status 1
podman info ``` host: arch: amd64 buildahVersion: 1.36.0 cgroupControllers: - cpu - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.10-1.fc40.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.10, commit: ' databaseBackend: boltdb distribution: distribution: fedora version: "40" eventLogger: journald freeLocks: 2048 idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 6.1.87-1 linkmode: dynamic logDriver: journald networkBackend: netavark networkBackendInfo: backend: netavark dns: package: aardvark-dns-1.10.0-1.fc40.x86_64 path: /usr/libexec/podman/aardvark-dns version: aardvark-dns 1.10.0 package: netavark-1.10.3-3.fc40.x86_64 path: /usr/libexec/podman/netavark version: netavark 1.10.3 ociRuntime: name: crun package: crun-1.15-1.fc40.x86_64 path: /usr/bin/crun version: |- crun version 1.15 commit: e6eacaf4034e84185fd8780ac9262bbf57082278 rundir: /run/user/1000/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL os: linux pasta: executable: /usr/bin/pasta package: passt-0^20240510.g7288448-1.fc40.x86_64 version: | pasta 0^20240510.g7288448-1.fc40.x86_64 Copyright Red Hat GNU General Public License, version 2 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: exists: true path: /run/user/1000/podman/podman.sock rootlessNetworkCmd: pasta security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.2-2.fc40.x86_64 version: |- slirp4netns version 1.2.2 commit: 0ee2d87523e906518d34a6b423271e4826f71faf libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.3 variant: "" plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io store: configFile: /home/user/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.force_mask: shared overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.13-1.fc40.x86_64 Version: |- fusermount3 version: 3.16.2 fuse-overlayfs: version 1.13-dev FUSE library version 3.16.2 using FUSE kernel interface version 7.38 graphRoot: /home/user/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Supports shifting: "true" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp runRoot: /run/user/1000/containers transientStore: false volumePath: /home/user/.local/share/containers/storage/volumes version: APIVersion: 5.1.0 GitCommit: "" GoVersion: go1.22.3 Os: linux OsArch: linux/amd64 Version: 5.1.0 ```
Rootfull build works ``` $ sudo podman build --no-cache -t testperms -f Containerfile.testperms STEP 1/4: FROM alpine:3.19 STEP 2/4: RUN mkdir -p /foo && chmod 777 /foo && chown 1234:1234 /foo && ls -la /foo total 8 drwxrwxrwx 2 1234 1234 4096 Jun 12 06:08 . dr-xr-xr-x 1 root root 4096 Jun 12 06:08 .. --> 244c412cda2e STEP 3/4: USER 1234 --> e0e0cf7a12d2 STEP 4/4: RUN ls -la /foo && touch /foo/bar total 8 drwxrwxrwx 2 1234 1234 4096 Jun 12 06:08 . dr-xr-xr-x 1 root root 4096 Jun 12 06:08 .. COMMIT testperms --> f38a01c90878 Successfully tagged localhost/testperms:latest ```
jiridanek commented 5 months ago

It's quite likely the same issue. I decided to report problems about a single configuration to make the bug report simpler, but I'm convinced that any force_mask that is less restrictive than 0777 will cause problems.

giuseppe commented 5 months ago

might be addressed by https://github.com/containers/fuse-overlayfs/pull/422

jiridanek commented 5 months ago

The tests in the PR do look promising!

legobeat commented 5 months ago

My issue is resolved after a system package upgrade (so might be different, given https://github.com/containers/fuse-overlayfs/pull/422 is not yet released).

$ sudo dnf list --installed | grep -E 'podman|container|overlay'
container-selinux.noarch                    2:2.232.1-1.fc40                  @updates
containerd.io.x86_64                        1.6.33-3.1.fc40                   @docker-ce-stable
containers-common.noarch                    5:0.59.1-2.fc40                   @updates
containers-common-extra.noarch              5:0.59.1-2.fc40                   @updates
fuse-overlayfs.x86_64                       1.13-1.fc40                       @fedora
overlayfs-tools.x86_64                      0-0.9.20200817git77bf3f7.fc40     @fedora
podman.x86_64                               5:5.1.1-1.fc40                    @updates
jiridanek commented 5 months ago

My issue is resolved after a system package upgrade (so might be different, given https://github.com/containers/fuse-overlayfs/pull/422 is not yet released).

@legobeat Did you do podman system reset after changing force_mask?

Initially, I just edited config to put force_mask back in, tried the reproducer command, and was about to celebrate, when I realized that it does not come into effect unless container storage is reset! So I did that, and bug started manifesting again.

jdanek@fedora:~$ vim ~/.config/containers/storage.conf 
jdanek@fedora:~$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
drwx------. 2 root root 104 Jun 20 13:28 /home/pepa/
drwx------. 2 pepa nobody 104 Jun 20 13:28 /home/pepa/
pepa
jdanek@fedora:~$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/jdanek/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y
 A "/home/jdanek/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.
jdanek@fedora:~$ podman run --rm -it registry.fedoraproject.org/fedora bash -c 'useradd pepa; ls -AlFd /home/pepa; chown pepa /home/pepa; ls -AlFd /home/pepa; sudo -u pepa ls /home'
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 0eb6f43df594 done   | 
Copying config 5c521474f1 done   | 
Writing manifest to image destination
drwx------. 2 root root 104 Jun 20 13:30 /home/pepa/
drwx------. 2 pepa nobody 104 Jun 20 13:30 /home/pepa/
sudo: unable to execute /usr/bin/ls: Permission denied
jdanek@fedora:~$
jiridanek commented 22 hours ago

My fancy images have fancy file types in them. When I tried this time, I got hit by