Nativu5 commented 1 year ago

Issue Description

I am experiencing a few issues while running a rootless container using the command:

podman run -it --network=host ubuntu:22.04 /bin/bash

The errors seemed to be related with permission settings on NFS.

Steps to reproduce the issue

Apt/dpkg: When I try to install or update software within the container using apt/dpkg, I encounter errors.

root@nfs-kernel:/# apt update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
...
Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [78.3 kB]
Fetched 28.2 MB in 8s (3451 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Download is performed unsandboxed as root as file '/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_jammy_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

User creation/access: In the container, creating a new user with useradd -m userinct and then attempting to switch to it using su - userinct, results in a "permission denied" error message. For some reason, the owner of the /home/userinct directory is being set as root incorrectly. Even after attempting to correct the owner with chown userinct:userinct -R /home/userinct, I still receive errors.

root@nfs-kernel:/# useradd -m userinct
root@nfs-kernel:/# su - userinct
su: warning: cannot change directory to /home/userinct: Permission denied
su: failed to execute /bin/sh: Permission denied
root@nfs-kernel:/# ls /home/ -l
total 0
drwxr-x---. 2 root root 93 Nov  1 05:51 userinct
root@nfs-kernel:/# chown userinct:userinct -R /home/userinct
root@nfs-kernel:/# ls /home/ -l
total 0
drwxr-x---. 2 userinct userinct 93 Nov  1 05:51 userinct
root@nfs-kernel:/# su - userinct
su: warning: cannot change directory to /home/userinct: Permission denied
su: failed to execute /bin/sh: Permission denied

Describe the results you received

As mentioned above.

Describe the results you expected

I have tested rootless containers in a non-NFS environment and there was no problem. I am asking if there is a way to get these containers work properly on NFS?

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 65953271fc1e506ac4eb890c645f3f75976973b4'
  cpuUtilization:
    idlePercent: 99.89
    systemPercent: 0.04
    userPercent: 0.07
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: '"centos"'
    version: "9"
  eventLogger: file
  freeLocks: 2047
  hostname: nfs-kernel
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 5.14.0-378.el9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 5322375168
  memTotal: 8058159104
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-3.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.10-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.10
      commit: c053c83c57551bca13ead8600237341818975974
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 6442446848
  swapTotal: 6442446848
  uptime: 138h 24m 4.00s (Approximately 5.75 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/nfsuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: "0700"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.el9.x86_64
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.12
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/nfsuser/.local/share/containers/storage
  graphRootAllocated: 37748736000
  graphRootUsed: 4777312256
  graphStatus:
    Backing Filesystem: nfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1001/containers
  transientStore: false
  volumePath: /home/nfsuser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1692961071
  BuiltTime: Fri Aug 25 18:57:51 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Environment:

OS: CentOS 9 Stream
Kernel version: 5.14.0-378.el9.x86_64
Podman version: 4.6.1
Home Directory of the user (nfsuser) is on NFS.

NFS Details:

fuse-overlayfs as the mounting program.
NFS protocol version is 4.2. Xattr on the NFS is tested and working properly.

nfsstat output:

[nfsuser@nfs-kernel ~]$ nfsstat -m
/home/nfsuser from 192.168.1.93:/nfs/nfsuser
Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.93,local_lock=none,addr=192.168.1.93

Additional information

Some of my investigation:

I am aware that the NFS was not supported back in 2019 (https://www.redhat.com/sysadmin/rootless-podman-nfs).
But I notice that there are some progresses (https://www.redhat.com/sysadmin/podman-rootless-overlay), which mentioned that Podman could utilized fuse-overlayfs and xattr on NFS to solve permission issues on NFS.
And another RedHat blog said that "Podman can now store data on any NFS server which supports extended attributes (xattrs)". (https://www.redhat.com/en/blog/rhel-9-delivers-latest-container-technologies-development-and-production#rhel9-nfs-container-storage)

Nativu5 commented 1 year ago

Given this information, it's my understanding that an NFS with Xattr support, coupled with fuse-overlayfs, should enable Podman to work correctly on NFS. However, despite these conditions being met in my case, I'm still facing permission issues.

Any help would be greatly appreciated.

rhatdan commented 1 year ago

@giuseppe PTAL

giuseppe commented 1 year ago

https://github.com/containers/storage/pull/1739 should solve the problem you have seen.

There are also some other upstream fixes in fuse-overlayfs that you might need

Nativu5 commented 1 year ago

containers/storage#1739 should solve the problem you have seen.

There are also some other upstream fixes in fuse-overlayfs that you might need

Thanks for the suggestion. I will give it a try and let you know the results.

Nativu5 commented 1 year ago

@giuseppe I have installed the bleeding-edge version of Podman (according to https://podman.io/docs/installation#installing-bleeding-edge-versions-of-podman) and I built the master branch of fuse-overlayfs. But these permissions issues persist.

Here is my test environment info:

 store:
  configFile: /home/nfsuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: "0700"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs-dev
      Package: Unknown
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/nfsuser/.local/share/containers/storage
  graphRootAllocated: 37748736000
  graphRootUsed: 6029312000
  graphStatus:
    Backing Filesystem: nfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1001/containers
  transientStore: false
  volumePath: /home/nfsuser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.0-dev-e622045af
  Built: 1698879181
  BuiltTime: Thu Nov  2 06:53:01 2023
  GitCommit: ""
  GoVersion: go1.19.13
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.0-dev-e622045af

giuseppe commented 1 year ago

you'd need to build it from source and vendor the last c/storage version

Nativu5 commented 1 year ago

you'd need to build it from source and vendor the last c/storage version

I have built the Podman from source, and provided it with latest containers/storage in go.mod (github.com/containers/storage v1.50.3-0.20231101112703-6e72f11598fb).

However, these problems are still reproducible.

store:
  configFile: /home/nfsuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: "0700"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs-dev
      Package: Unknown
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/nfsuser/.local/share/containers/storage
  graphRootAllocated: 37748736000
  graphRootUsed: 8726249472
  graphStatus:
    Backing Filesystem: nfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1001/containers
  transientStore: false
  volumePath: /home/nfsuser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.0-dev
  Built: 1698925657
  BuiltTime: Thu Nov  2 19:47:37 2023
  GitCommit: 3ef2f139623ce213cd88117799c87ade4cc093a5
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.0-dev

giuseppe commented 1 year ago

Can you try force_mask=755?

NFS doesn't honor CAP_DAC_OVERRIDE in a user namespace and that might cause the problem you are seeing

Nativu5 commented 1 year ago

Can you try force_mask=755?

NFS doesn't honor CAP_DAC_OVERRIDE in a user namespace and that might cause the problem you are seeing

I have set force_mask to 0755 but the issues are not resolved.

The good news is, now I can add a new user in the container and switch to it. But the permission of that user's home directory is still broken.

root@nfs-kernel:/# useradd -m test
root@nfs-kernel:/# ls -l /home/
total 0
drwxr-x---. 2 root root 93 Nov  2 14:37 test
root@nfs-kernel:/# su - test
su: warning: cannot change directory to /home/test: Permission denied
$ whoami
test

Meanwhile, the apt/dpkg is still unusable. When I tried to install openssh-client using apt, errors seemed to jump out after these:

Setting up openssh-client (1:8.9p1-3ubuntu0.4) ...
chmod: changing permissions of '/usr/bin/ssh-agent': Value too large for defined data type
dpkg: error processing package openssh-client (--configure):
 installed openssh-client package post-installation script subprocess returned error exit status 1

Shadow8472 commented 1 year ago

I believe I'm having this issue too.

20519

github-actions[bot] commented 11 months ago

A friendly reminder that this issue had no activity for 30 days.

pashok2398 commented 11 months ago

+1

containers / podman

Permission Issues while running a Rootless Container on CentOS 9 Stream with NFS Home Directory #20551

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

20519