Permission error on bind mount when using provider applehv

[tnk4on]

Issue Description

When creating a machine with CONTAINERS_MACHINE_PROVIDER=applehv on a macOS's Podman machine, the bind mounted directory is not available due to permission error.

Steps to reproduce the issue

% podman machine rm -f
% export CONTAINERS_MACHINE_PROVIDER=qemu
% podman machine init; podman machine start
Extracting compressed file: podman-machine-default_fedora-coreos-39.20231204.2.1-qemu.aarch64.qcow2: done
...
% podman run --rm -v $PWD:$PWD -w $PWD ubi9/ubi touch test.txt;ls -ld test.txt
-rw-r--r--  1 shtanaka  staff  0 Dec 25 00:02 test.txt

### reset environment
% podman machine rm -f
% rm -f test.txt 

% export CONTAINERS_MACHINE_PROVIDER=applehv
% podman machine init; podman machine start
Extracting compressed file: podman-machine-default_fedora-coreos-39.20231204.2.1-applehv.aarch64.raw: done
...
% podman run --rm -v $PWD:$PWD -w $PWD ubi9/ubi touch test.txt;ls -ld test.txt
touch: cannot touch 'test.txt': Permission denied
ls: test.txt: No such file or directory

Describe the results you received

touch: cannot touch 'test.txt': Permission denied
ls: test.txt: No such file or directory

Describe the results you expected

-rw-r--r--  1 shtanaka  staff  0 Dec 25 00:02 test.txt

podman info output

% podman info
host:
  arch: arm64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 90.07
    systemPercent: 5.3
    userPercent: 4.62
  cpus: 5
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.6.3-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 120045568
  memTotal: 1982873600
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.12-1.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231119.g4f1709d-1.fc39.aarch64
    version: |
      pasta 0^20231119.g4f1709d-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 31.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 2591690752
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762633
  BuiltTime: Tue Oct 31 23:30:33 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/arm64
  Version: 4.7.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

% sw_vers 
ProductName:            macOS
ProductVersion:         14.2.1
BuildVersion:           23C71

% vfkit -v
vfkit version: 0.5.0

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Did you try to add :Z to the volume? The machine has selinux enabled so it needs the right labels.

[tnk4on]

Thanks for the advice! The results are as follows.

In qemu I get an error with :Z and the file is not created.

% podman run --rm -v $PWD:$PWD:Z -w $PWD ubi9/ubi touch test.txt;ls -ld test.txt
Error: preparing container 54b3a4934d05ead16381bd9957af9efe1666ae09ee0194074d579caa07b605da for attach: lsetxattr /Users/shtanaka/tmp/test: operation not supported
ls: test.txt: No such file or directory
% podman machine ssh
$ ls -lZd /Users/shtanaka/tmp/test
drwxr-xr-x. 2 core games system_u:object_r:nfs_t:s0 64 Jan  4 00:23 /Users/shtanaka/tmp/test

Also, in applehv, I get a Permission denied error, but the file is successfully created.

% podman run --rm -v $PWD:$PWD:Z -w $PWD ubi9/ubi touch test.txt;ls -ld test.txt
touch: cannot touch 'test.txt': Permission denied
-rw-r--r--  1 shtanaka  staff  0  1  4 00:26 test.txt
% podman machine ssh
$ ls -ldZ /Users/shtanaka/tmp/test
drwxr-xr-x. 3 core core system_u:object_r:container_file_t:s0:c517,c931 96 Jan  4 00:26 /Users/shtanaka/tmp/test

In applehv, there is no consistency between the error message and the actual behavior. However, I was surprised because I thought relabeling was not available on macOS !

[Luap99]

relabelling is not available with 9p which is used with qemu, for applehv we switched to the much better performing virtiofs which apparently supports xattrs so labelling works correctly.

https://github.com/containers/podman/pull/20746 changed the code to no longer report an error when :Z is given for file systems that do not support selinux labelling. So going forward it should be safe to always use :Z as with qemu it would juts be ignored. This change would land in 5.0 or maybe even 4.9.

[towertop]

Reach here with same issue. I tried the Z option but the permission error stills. That prevented my VSCode from working in created containers.

My .config/containers/containers.conf content:

[containers]

[engine]

[machine]
  volumes = ["/Users:/Users:Z", "/private:/private:Z", "/var/folders:/var/folders:Z"]
  provider="applehv"

[network]

[secrets]

[configmaps]

[farms]

The machine started with vfkit also suffered unresponsive trouble when try to stop or remove it. I need to kill the vfkit process from MacOS monitor. There have been issues tracking this.

[Luap99]

the :Z or :z must be set one the podman run command not for machine init

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[tnk4on]

fixed by https://github.com/containers/podman/pull/21297 and https://github.com/containers/podman/pull/21320 .