Rootless Podman 3.0 RC1 on WSL Fedora34 image- can't run as rootless user

[PavelSosin-320]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman 3.0 RC1 on WSL Fedora34 distro doesn't work in the rootless mode. info Steps to reproduce the issue:

1. Create a new WSL2 Fedora34 following Fedora magazine guidelines.
2. Install the recent release of genie-systemd for Fedora34 to run the distro as bare-metall Linux Fedora 34
3. Run the distro using wsl -d Fedora34 -u root genie -s
4. Install Podman 3.0 RC1 build from podman-3.0.0-1.fc34.x86_64.rpm and podman-plugins-3.0.0-1.fc34.x86_64.rpm
5. Resolve missed dependencies: error: Failed dependencies: conmon >= 2:2.0.16-1 is needed by podman-2:3.0.0-0.1.rc1.fc33.x86_64 containernetworking-plugins >= 0.8.6-1 is needed by podman-2:3.0.0-0.1.rc1.fc33.x86_64 containers-common >= 1:1.2.0-9 is needed by podman-2:3.0.0-0.1.rc1.fc33.x86_64 oci-runtime is needed by podman-2:3.0.0-0.1.rc1.fc33.x86_64
6. Test Podman as a root user -> everything OK except wrong cgroups version detection: The Kernel 5.9 and distro obviously support V2 but V1 is shown ( The same issue as in Fedora 33 and Ubuntu 2010 ).
7. Login as non-root user using wsl -d Fedora34 -u aRootlessUser genie -l
8. Try Podman info

Describe the results you received: ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve Error: Cannot connect to the Podman socket, make sure there is a Podman REST API service running.: cannot setup namespace using newuidmap: exit status 1

Describe the results you expected: Same as for root user except rootless: true.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.15.6
Built:        Mon Jan 18 18:07:50 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.15.6
Built:        Mon Jan 18 18:07:50 2021
OS/Arch:      linux/amd64
[root@MSI-wsl ~]# podman info
host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.22-0.9.dev.git2fbeb9f.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.22-dev, commit: 87878e20b6fe74f377214092876c0a3745fe221b'
  cpus: 4
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: MSI-wsl
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.91-microsoft-standard-WSL2
  linkmode: dynamic
  memFree: 11636736000
  memTotal: 12559527936
  ociRuntime:
    name: crun
    package: crun-0.18-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 3221225472
  swapTotal: 3221225472
  uptime: 6h 24m 9.78s (Approximately 0.25 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1610986070
  BuiltTime: Mon Jan 18 18:07:50 2021
  GitCommit: ""
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 3.0.0-rc1

[root@MSI-wsl ~]# podman version
Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.15.6
Built:        Mon Jan 18 18:07:50 2021
OS/Arch:      linux/amd64
[

Package info (e.g. output of rpm -q podman or apt list podman): podman-3.0.0-0.1.rc1.fc33.x86_64 is it wrong rpm?

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): If the issue's maintaner needs the distro installed without the headache I can publish it via Microsoft OneDrive as tar to be installed using wsl --import in 5 minutes.

[vrothberg]

Can you try with the latest version? You are using release candidate 1.

[PavelSosin-320]

I suppose something wrong with the Fedora build system. dnf install Podman from Fedora 34 brings Podman version 3.0-dev with precisely the same issue. It was my starting point. Then I took rpm from the Fedora build system. I did it for Podman 3.0 RC1 for Fedora33 and it was mainly OK except for the wrong cgroups version. Unfortunately, my Fedora 33 was unrecoverably corrupted by OpenSSH server. This is known issue. I found that in Fedora 34 systemd is also in the RC2 phase: systemctl --version systemd 248 (v248~rc2-1.fc34) +PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

[PavelSosin-320]

And Kernel startup log: [root@MSI-wsl ~]# journalctl -b -- Journal begins at Wed 2021-02-24 18:51:39 IST, ends at Thu 2021-02-25 20:08:49 IST. -- Feb 25 19:32:03 MSI-wsl kernel: Linux version 5.4.91-microsoft-standard-WSL2 (oe-user@oe-host) ( gcc version 9.3.0 (GCC)) #1 SMP Mon Jan 25 18:39:31 UTC 2021 Feb 25 19:32:03 MSI-wsl kernel: Command line: initrd=\initrd.img panic=-1 nr_cpus=4 swiotlb=force console=ttyS0,115200 d ebug pty.legacy_count=0 systemd.unified_cgroup_hierarchy=1 <----------- ??? FYI! The story of Docker, Fedora, and groups V2 How to install and use Docker on Fedora . But it is solved in the latest release.

[vrothberg]

Thanks, @PavelSosin-320! There are a number of issues at the moment in Fedora 34. @giuseppe @lsm5, I assume Podman and others are currently blocked on the glibc bump and seccomp farts, right?

[PavelSosin-320]

@vrothberg You mentioned seccomp. What I has read about seccomp it provides syscalls auditing at the node level. Is it applicable on use-case when Podman runs inside lightweight containers like WSL or other kind of software machines? If Microsoft decided to not provide any WSL system calls auditing to the LXSS VM Kernel such level of security is useless. It needs its own configuration. It is overkilling demand for WSL user. Could it be ignored by configuration or environment ? In the current version Podman info shows path: /run/podman/podman.sock security: apparmorEnabled: false <---- correct capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true <--------- Wrong??? selinuxEnabled: false <---------- correct

[PavelSosin-320]

The same in podman version Version: 3.0.1 API Version: 3.0.0 Go Version: go1.16 Built: Mon Feb 22 16:08:57 2021 OS/Arch: linux/amd64 dnf info podman Last metadata expiration check: 0:16:11 ago on Sun Feb 28 08:38:45 2021. Installed Packages Name : podman Epoch : 2 Version : 3.0.1 Release : 1.fc34 Architecture : x86_64 Size : 44 M Source : podman-3.0.1-1.fc34.src.rpm Repository : @System From repo : fedora Summary : Manage Pods, Containers and Container Images URL : https://podman.io/ License : ASL 2.0 Buildah tolerates this issue: WARN error running newgidmap: exit status 1: newgidmap: write to gid_map failed: Operation not permitted WARN falling back to single mapping WARN error running newuidmap: exit status 1: newuidmap: write to uid_map failed: Operation not permitted WARN falling back to single mapping { "host": { "CgroupVersion": "v1", "Distribution": { "distribution": "fedora", "version": "34" }, "MemFree": 11492716544, "MemTotal": 12559527936, "OCIRuntime": "crun", "SwapFree": 3221225472, "SwapTotal": 3221225472, "arch": "amd64", "cpus": 4, "hostname": "MSI-wsl", "kernel": "5.4.91-microsoft-standard-WSL2", "os": "linux", "rootless": true, "uptime": "8h 26m 9.61s (Approximately 0.33 days)" }, "store": { "ContainerStore": { "number": 0 }, "GraphDriverName": "overlay", "GraphOptions": [ "overlay.mount_program=/usr/bin/fuse-overlayfs" ], "GraphRoot": "/home/pavel/.local/share/containers/storage", "GraphStatus": { "Backing Filesystem": "extfs", "Native Overlay Diff": "false", "Supports d_type": "true", "Using metacopy": "false" }, "ImageStore": { "number": 0 }, "RunRoot": "/run/user/1000/containers" } }

[PavelSosin-320]

Rootless Docker solution . I expect the same from Podman

[PavelSosin-320]

It also happens with buildah from: Error committing the finished image: error adding layer with blob "sha256:a591faa84ab05242a17131e396a336da172b0e1ec66d921c9f130b7c4c24586d": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid: lchown /var/spool/mail: invalid argument ERRO exit status 125 [pavel@MSI-wsl ~]$ cat /etc/subuid pavel:100000:65536 [pavel@MSI-wsl ~]$ cat /etc/subgid pavel:100000:65536 [pavel@MSI-wsl ~]$

And looks like related to Rootless Podman restrictions -cgroups V2 ! post from Dan Walsh In other words, it can't work if cgroups version is not recognized correctly! Pleeease, correct this issue!

[vrothberg]

Can you try running Fedora 33? As I mentioned, Fedora 34 has seccomp issues.

[PavelSosin-320]

I tested Podman 3 RC1 with Fedora33. It has no such issue. But I believe that the route cause is not in the Fedora version. I use Fedora WSL image. Natively, the are some architectural differences between Linux running on the hardware and Linux running on the software machine. I'm searching a way how to mitigate these differences. For example, the /boot/ folder is empty in my machine. VM is not booted as a server. The Microsoft WSL Kernel is delivered separately from distros and shared among all distros. Anyway, I expect that Kernel 5.4.9 with Fedora 34 distro can support Podman 3. Docker mitigates such architectural nuances. The security capabilities can be passed to the container via Docker security profile option. It creates a palpable gap between Docker and Podman features. I expect that by default all security options in the Podman configuration should be set to false in the rootless mode. Rootless user has no authorization to make proper security configuration.

[vrothberg]

That confirms it's the seccomp issue in F34 (https://bugzilla.redhat.com/show_bug.cgi?id=1927635).

The security capabilities can be passed to the container via Docker security profile option. It creates a palpable gap between Docker and Podman features

Podman allows for passing these options in the very same way as Docker. I don't see any gap.

I expect that by default all security options in the Podman configuration should be set to false in the rootless mode. Rootless user has no authorization to make proper security configuration.

Rootless users have for sure the power to make proper security configurations. Disabling them by default would be dangerous.

[PavelSosin-320]

@vrothberg I looked into Bugzilla issue requesting libseccomp 2.5. I already have this version: dnf info libseccomp.x86_64 Last metadata expiration check: 2:07:51 ago on Tue Mar 2 17:41:01 2021. Installed Packages Name : libseccomp Version : 2.5.0 Release : 4.fc34 Architecture : x86_64 Size : 174 k Source : libseccomp-2.5.0-4.fc34.src.rpm Repository : @System From repo : fedora Summary : Enhanced seccomp library URL : https://github.com/seccomp/libseccomp License : LGPLv2 Description : The libseccomp library provides an easy to use interface to the Linux Kernel's : syscall filtering mechanism, seccomp. The libseccomp API allows an application : to specify which syscalls, and optionally which syscall arguments, the : application is allowed to execute, all of which are enforced by the Linux : Kernel.

[root@MSI-wsl ~]# dnf install libseccomp.x86_64 Last metadata expiration check: 2:08:14 ago on Tue Mar 2 17:41:01 2021. Package libseccomp-2.5.0-4.fc34.x86_64 is already installed. Dependencies resolved. I am afraid that the issue is in the Microsoft-made Kernel 5.4.9 that doesn't support seccomp because it is Microsoft security.

[PavelSosin-320]

@vrothberg I'm afraid I'm right: auditctl -s Error - audit support not in kernel Cannot open netlink audit socket. Microsoft Kernel means "Microsoft security"

[vrothberg]

Likely more seccomp work needed in F34. Given that F33 works and that Podman is always using seccomp, I doubt that it's the Microsoft kernel. Audit is not seccomp.

[mheon]

I think we'd run into more serious issues if the kernel didn't have Seccomp - runc, at least, wouldn't be able to make containers at all (root or rootless). crun could be different, but I don't think so.

[PavelSosin-320]

@vrothberg Thanks a lot !!!!!!!!!!!!!!!! I'm back on the track after upgrading secomp and secomp_static to 2.5, running with audit=1, and applying Podman 3.0.1 has started to work on Fedora34 WSL image for rootless users. It was expected because of fedora34 beta release is planned to the middle of March and the final release in April. If somebody wants to use Podman for development on the "pupil's laptop" with OEM Home Windows I can upload my WSL distro to MS One drive and send a share.