Permission denied extracting to etc.defaults/shadow within a bind mount on MacOS, works with Docker

wgordon17 commented 3 months ago

Issue Description

I've read through everything in the Troubleshooting Guide to no avail 😞

I'm trying to launch a Virtual Synology DSM image, which uses some containerized virtual machine voodoo 😉 by running Qemu in a container, and then using Qemu to manage the DSM VM. It certainly isn't pretty, but it works just fine with Docker.

Steps to reproduce the issue

I'm running rootful podman
mkdir DSM_TESTING
sudo podman run -it --rm -p 5001:5000 -p 2222:22 -v $PWD/DSM_TESTING:/storage:Z --userns keep-id --security-opt label=disable --privileged --cap-add NET_ADMIN -e ALLOCATE=N --stop-timeout 120 vdsm/virtual-dsm

Describe the results you received

This is the output of podman

❯ Starting Virtual DSM for Docker v7.15...
❯ For support visit https://github.com/vdsm/virtual-dsm
❯ CPU:   | RAM: 3/4 GB | DISK: 19 GB (fuseblk) | HOST: 6.7.7-200.fc39.aarch64...

❯ Warning: the filesystem of /storage is FUSE, this extra layer will negatively affect performance!
❯ Install: Downloading DSM_VirtualDSM_69057.pat...
/DSM_VirtualDSM_69057.pat                100%[===============================================================================>] 346.98M  17.3MB/s    in 20s
❯ Install: Extracting downloaded image...
❯ Install: Preparing system partition...
❯ Install: Extracting system partition...
tar: etc.defaults/shadow: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors
❯ ERROR: Status 2 while: fakeroot -- bash -c "set -Eeu;        tar xpfJ $HDA.txz --absolute-names --skip-old-files -C $MOUNT/;        printf '%b%s%b' '\E[1;34m❯ \E[1;36m' 'Install: $MSG' '\E[0m\n';        mke2fs -q -t ext4 -b 4096 -d $MOUNT/ -L $LABEL -F -E offset=$OFFSET $SYSTEM $NUMBLOCKS" (line 391/10)

The specific error message seems to be coming from tar: etc.defaults/shadow: Cannot open: Permission denied

Describe the results you expected

But everything succeeds with docker

$ docker run -it --rm -p 5001:5000 -p 2222:22 -v (pwd)/DSM_TESTING2:/storage --privileged --cap-add NET_ADMIN -e ALLOCATE=N --stop-timeout 120 vdsm/virtual-dsm

❯ Starting Virtual DSM for Docker v7.15...
❯ For support visit https://github.com/vdsm/virtual-dsm
❯ CPU:   | RAM: 7/8 GB | DISK: 15 GB (UNKNOWN (0x6a656a63)) | HOST: 6.6.31-linuxkit...

❯ Install: Downloading installer...
❯ Install: Downloading DSM_VirtualDSM_69057.pat...
/DSM_VirtualDSM_69057.pat                100%[===============================================================================>] 346.98M  17.1MB/s    in 19s
❯ Install: Extracting downloaded image...
❯ Install: Preparing system partition...
❯ Install: Extracting system partition...
❯ Install: Installing system partition...
❯ Creating a 16G growable disk image in raw format...
❯ Warning: your container IP starts with 172.17.* which will cause conflicts when you install the Container Manager package inside DSM!
❯ Warning: your CPU architecture is ARM64 and cannot provide KVM acceleration for x64 instructions, this will cause a major loss of performance.
❯ Booting Virtual DSM using QEMU v8.2.4...

podman info output

host:
  arch: arm64
  buildahVersion: 1.35.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 86.78
    systemPercent: 2.04
    userPercent: 11.18
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.7.7-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1827328000
  memTotal: 3798147072
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.20240312103946045036.main.18.g8377c0a.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: netavark-1.10.1-1.20240319111419242122.main.46.gcc3f35d.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0-dev
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.20240302220834691516.main.10.g64ee22c.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: feb70bc2ab11944a6443e4f5d2eb96a22f186b80
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.aarch64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: ""
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 15m 46.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 13353594880
  graphRootUsed: 5653868544
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.0-dev-8a643c243
  Built: 1710720000
  BuiltTime: Sun Mar 17 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.0-dev-8a643c243

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Running on Apple M1 Pro, Sonoma 14.5

 $ podman version

Client:       Podman Engine
Version:      5.1.1
API Version:  5.1.1
Go Version:   go1.22.3
Git Commit:   bda6eb03dcbcf12a5b7ae004c1240e38dd056d24
Built:        Tue Jun  4 15:54:07 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.0-dev-8a643c243
API Version:  5.0.0-dev-8a643c243
Go Version:   go1.21.8
Built:        Sun Mar 17 20:00:00 2024
OS/Arch:      linux/arm64

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Luap99 commented 3 months ago

Don't use --userns keep-id when running as root unless you know what this means. So I suggest you first try without that. I am not sure if that is related or now but it certainly does not help.

wgordon17 commented 3 months ago

Thanks @Luap99! It's a consequence of just blindly following suggestions in the Troubleshooting Guide, https://github.com/containers/podman/blob/main/troubleshooting.md#2-cant-use-volume-mount-get-permission-denied 😉

I tried again without --userns keep-id, and no difference in output 😞

baude commented 3 months ago

what is in that directory. Any chance there are files with unusual permissions or possibly read-only content?

wgordon17 commented 3 months ago

So, the host directory, DSM_TESTING, is initially completely empty. It was created for the purpose of this container. I'm literally just running mkdir DSM_TESTING right before I run the podman run command.

rhatdan commented 3 months ago

Are you on a MAC and running sudo podman?

wgordon17 commented 3 months ago

@rhatdan That is accurate 😅 Just throwing everything to the wall to see what sticks ¯\_(ツ)_/¯

rhatdan commented 3 months ago

Most likely what is happening is a UID is attempting to be stored on the MAC that is not the users UID. Looks etc/shadow is the culprit.

wgordon17 commented 3 months ago

Looking at the file directory when created by docker, I'm seeing

----------    1 wgordon  staff     0B Jun 17 11:23 shadow

Which seems to be fine?

wgordon17 commented 3 months ago

I did also attempt an explicit --userns keep-id:uid=501,gid=20 previously (where 501 matches UID wgordon, and 20 matches GID staff). I didn't make any difference ¯\_(ツ)_/¯

Luap99 commented 3 months ago

That means that nobody (not even the owner) has access to the file. Only a process with CAP_DAC_OVERRIDE on linux (or typically root) can access such a file. Podman machine always runs as user on the macos host so it would not have permissions to use such file on the host system I would assume. You can try to read/write this file as you macos user it will not work only as root it can work there.

Therefore I don't really see how this could work with podman machine. You could try it without a host mapped volume.

rhatdan commented 3 months ago

Could you try this with vfkit and virtiofsd, if you are continuing to use qemu for the podman machine?

Luap99 commented 3 months ago

Whatever you do inside the VM shouldn't matter, this can be trivially reproduced by just using podman machine ssh, then change to a mounted volume, i.e. cd /Users/<username>/.

core@localhost:/Users/paul$ touch test
core@localhost:/Users/paul$ chmod 0000 test
core@localhost:/Users/paul$ ls test
test
core@localhost:/Users/paul$ ls -l test
ls: test: Permission denied
----------. 1 core core 0 Jun 26 11:15 test
core@localhost:/Users/paul$ cat test
cat: test: Permission denied
core@localhost:/Users/paul$ sudo cat test
cat: test: Permission denied

Only on the macOS host you can successfully read/write the file with sudo. And because we launch the VM with user privs I don't assume the viriofs endpoint on the host is not run as root thus it logically cannot read such a file which means no one in the VM can use this file.

wgordon17 commented 2 months ago

Only on the macOS host you can successfully read/write the file with sudo. And because we launch the VM with user privs I don't assume the viriofs endpoint on the host is not run as root thus it logically cannot read such a file which means no one in the VM can use this file.

Do you have any concrete suggestions on something I could do to overcome this issue? Or is using docker my only solution for now?

Luap99 commented 2 months ago

Do not use a host volume mount point for the extraction.

containers / podman