[Bug]: unable to build or run containers

Issue Description

I'm unable to build or run rootless containers. I've found similar issues already reported, but none of the fixes work.

Steps to reproduce the issue

Steps to reproduce the issue:

podman run --rm -it fedora

Describe the results you received

Describe the results you received:

$ podman run --rm -it fedora
Error: runc: runc create failed: unable to start container process: error during container init: error setting rlimits for ready process: error setting rlimit type 7: operation not permitted: OCI permission denied

full debug log

``` level=info msg="podman filtering at log level debug" level=debug msg="Called run.PersistentPreRunE(podman --log-level=debug run --rm -it fedora)" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" level=debug msg="Merged system config \"/etc/containers/containers.conf\"" level=debug msg="Using conmon: \"/usr/bin/conmon\"" level=debug msg="Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db" level=debug msg="Overriding run root \"/run/user/2181/containers\" with \"/tmp/podman-run-2181/containers\" from database" level=debug msg="Overriding tmp dir \"/run/user/2181/libpod/tmp\" with \"/tmp/podman-run-2181/libpod/tmp\" from database" level=debug msg="Using graph driver overlay" level=debug msg="Using graph root /home/user/.local/share/containers/storage" level=debug msg="Using run root /tmp/podman-run-2181/containers" level=debug msg="Using static dir /home/user/.local/share/containers/storage/libpod" level=debug msg="Using tmp dir /tmp/podman-run-2181/libpod/tmp" level=debug msg="Using volume path /home/user/.local/share/containers/storage/volumes" level=debug msg="Set libpod namespace to \"\"" level=debug msg="Not configuring container store" level=debug msg="Initializing event backend file" level=debug msg="Configured OCI runtime runj initialization failed: cannot stat OCI runtime runj path: stat /usr/local/bin/runj: permission denied" level=debug msg="Configured OCI runtime kata initialization failed: cannot stat OCI runtime kata path: stat /usr/local/bin/kata-runtime: permission denied" level=debug msg="Configured OCI runtime runsc initialization failed: cannot stat OCI runtime runsc path: stat /usr/local/bin/runsc: permission denied" level=debug msg="Configured OCI runtime krun initialization failed: cannot stat OCI runtime krun path: stat /usr/local/bin/krun: permission denied" level=debug msg="Using OCI runtime \"/usr/bin/runc\"" level=info msg="Setting parallel job count to 121" level=info msg="podman filtering at log level debug" level=debug msg="Called run.PersistentPreRunE(podman --log-level=debug run --rm -it fedora)" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" level=debug msg="Merged system config \"/etc/containers/containers.conf\"" level=debug msg="Using conmon: \"/usr/bin/conmon\"" level=debug msg="Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db" level=debug msg="Overriding run root \"/run/user/2181/containers\" with \"/tmp/podman-run-2181/containers\" from database" level=debug msg="Overriding tmp dir \"/run/user/2181/libpod/tmp\" with \"/tmp/podman-run-2181/libpod/tmp\" from database" level=debug msg="Using graph driver overlay" level=debug msg="Using graph root /home/user/.local/share/containers/storage" level=debug msg="Using run root /tmp/podman-run-2181/containers" level=debug msg="Using static dir /home/user/.local/share/containers/storage/libpod" level=debug msg="Using tmp dir /tmp/podman-run-2181/libpod/tmp" level=debug msg="Using volume path /home/user/.local/share/containers/storage/volumes" level=debug msg="Set libpod namespace to \"\"" level=debug msg="[graphdriver] trying provided driver \"overlay\"" level=debug msg="Cached value indicated that overlay is supported" level=debug msg="Cached value indicated that overlay is supported" level=debug msg="Cached value indicated that metacopy is not being used" level=debug msg="Cached value indicated that native-diff is usable" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false" level=debug msg="Initializing event backend file" level=debug msg="Configured OCI runtime krun initialization failed: cannot stat OCI runtime krun path: stat /usr/local/bin/krun: permission denied" level=debug msg="Configured OCI runtime runj initialization failed: cannot stat OCI runtime runj path: stat /usr/local/bin/runj: permission denied" level=debug msg="Configured OCI runtime kata initialization failed: cannot stat OCI runtime kata path: stat /usr/local/bin/kata-runtime: permission denied" level=debug msg="Configured OCI runtime runsc initialization failed: cannot stat OCI runtime runsc path: stat /usr/local/bin/runsc: permission denied" level=debug msg="Using OCI runtime \"/usr/bin/runc\"" level=info msg="Setting parallel job count to 121" level=debug msg="Failed to add podman to systemd sandbox cgroup: dial unix @/tmp/dbus-GP29DcK06N: connect: connection refused" level=debug msg="Pulling image fedora (policy: missing)" level=debug msg="Looking up image \"fedora\" in local containers storage" level=debug msg="Normalized platform linux/amd64 to {amd64 linux [] }" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\"" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\"" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/001-rhel-shortnames.conf\"" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf\"" level=debug msg="Trying \"registry.fedoraproject.org/fedora:latest\" ..." level=debug msg="parsed reference into \"[overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage ([overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Looking up image \"registry.fedoraproject.org/fedora:latest\" in local containers storage" level=debug msg="Normalized platform linux/amd64 to {amd64 linux [] }" level=debug msg="Trying \"registry.fedoraproject.org/fedora:latest\" ..." level=debug msg="parsed reference into \"[overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Found image \"registry.fedoraproject.org/fedora:latest\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage" level=debug msg="Found image \"registry.fedoraproject.org/fedora:latest\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage ([overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Looking up image \"fedora\" in local containers storage" level=debug msg="Normalized platform linux/amd64 to {amd64 linux [] }" level=debug msg="Trying \"registry.fedoraproject.org/fedora:latest\" ..." level=debug msg="parsed reference into \"[overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage ([overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" level=debug msg="using systemd mode: false" level=debug msg="No hostname set; container's hostname will default to runtime default" level=debug msg="Loading seccomp profile from \"/usr/share/containers/seccomp.json\"" level=debug msg="Allocated lock 1 for container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" level=debug msg="parsed reference into \"[overlay@/home/user/.local/share/containers/storage+/tmp/podman-run-2181/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" level=debug msg="Cached value indicated that idmapped mounts for overlay are not supported" level=debug msg="Check for idmapped mounts support " level=debug msg="Created container \"86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2\"" level=debug msg="Container \"86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2\" has work directory \"/home/user/.local/share/containers/storage/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata\"" level=debug msg="Container \"86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2\" has run directory \"/tmp/podman-run-2181/containers/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata\"" level=debug msg="Handling terminal attach" level=debug msg="[graphdriver] trying provided driver \"overlay\"" " level=debug msg="Cached value indicated that overlay is supported" " level=debug msg="Made network namespace at /run/user/2181/netns/netns-0fc289bb-4b4d-1533-e3a0-d8dc6303f594 for container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" " level=debug msg="Cached value indicated that overlay is supported" " level=debug msg="Cached value indicated that metacopy is not being used" " level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false" " level=debug msg="Cached value indicated that volatile is being used" " level=debug msg="overlay: mount_data=lowerdir=/home/user/.local/share/containers/storage/overlay/l/U2TKM3DNCDYWXRHJWWR2KC6MBQ,upperdir=/home/user/.local/share/containers/storage/overlay/b9b07c54dbca2f1029488c66c00b778cd319a58dd6872df272992ab70e13e0c6/diff,workdir=/home/user/.local/share/containers/storage/overlay/b9b07c54dbca2f1029488c66c00b778cd319a58dd6872df272992ab70e13e0c6/work,,userxattr,volatile,context=\"system_u:object_r:container_file_t:s0:c188,c956\"" " level=debug msg="slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/2181/netns/netns-0fc289bb-4b4d-1533-e3a0-d8dc6303f594 tap0" " level=debug msg="Mounted container \"86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2\" at \"/home/user/.local/share/containers/storage/overlay/b9b07c54dbca2f1029488c66c00b778cd319a58dd6872df272992ab70e13e0c6/merged\"" " level=debug msg="Created root filesystem for container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2 at /home/user/.local/share/containers/storage/overlay/b9b07c54dbca2f1029488c66c00b778cd319a58dd6872df272992ab70e13e0c6/merged" " level=debug msg="/etc/system-fips does not exist on host, not mounting FIPS mode subscription" " level=debug msg="reading hooks from /usr/share/containers/oci/hooks.d" " level=debug msg="Workdir \"/\" resolved to host path \"/home/user/.local/share/containers/storage/overlay/b9b07c54dbca2f1029488c66c00b778cd319a58dd6872df272992ab70e13e0c6/merged\"" " level=debug msg="Created OCI spec for container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2 at /home/user/.local/share/containers/storage/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata/config.json" " level=debug msg="/usr/bin/conmon messages will be logged to syslog" " level=debug msg="running conmon: /usr/bin/conmon" args="[--api-version 1 -c 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2 -u 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2 -r /usr/bin/runc -b /home/user/.local/share/containers/storage/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata -p /tmp/podman-run-2181/containers/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata/pidfile -n romantic_jang --exit-dir /tmp/podman-run-2181/libpod/tmp/exits --full-attach -l k8s-file:/home/user/.local/share/containers/storage/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /tmp/podman-run-2181/containers/overlay-containers/86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/user/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/podman-run-2181/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /tmp/podman-run-2181/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /home/user/.local/share/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2]" " level=info msg="Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpuset: mkdir /sys/fs/cgroup/cpuset/conmon: permission denied" " level=debug msg="Received: -1" " level=debug msg="Cleaning up container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" " level=debug msg="Tearing down network namespace at /run/user/2181/netns/netns-0fc289bb-4b4d-1533-e3a0-d8dc6303f594 for container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" " level=debug msg="Unmounted container \"86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2\"" " level=debug msg="Removing container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" level=debug msg="Cleaning up container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" level=debug msg="Network is already cleaned up, skipping..." level=debug msg="Container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2 storage is already unmounted, skipping..." level=debug msg="Removing all exec sessions for container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2" level=debug msg="Container 86c9e6087bf22e92cf03c69759201d1f79ad8e18597660c30fc1aca98ebe78d2 storage is already unmounted, skipping..." level=debug msg="ExitCode msg: \"runc: time=\\\"2023-02-03t14:43:45-05:00\\\" level=warning msg=\\\"unable to get oom kill count\\\" error=\\\"no directory specified for memory.oom_control\\\"\\ntime=\\\"2023-02-03t14:43:45-05:00\\\" level=error msg=\\\"runc create failed: unable to start container process: error during container init: error setting rlimits for ready process: error setting rlimit type 7: operation not permitted\\\": oci permission denied\"" 43:45-05:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control" level=error msg="runc create failed: unable to start container process: error during container init: error setting rlimits for ready process: error setting rlimit type 7: operation not permitted": OCI permission denied ```

Describe the results you expected

container runs normally

podman info output

$ podman version
Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Mon Dec 12 06:41:56 2022
OS/Arch:      linux/amd64

full podman info --debug

``` host: arch: amd64 buildahVersion: 1.27.3 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.4-1.module+el8.7.0+17498+a7f63b89.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.4, commit: 419a7c7817d7da098aa7648abddea5014f593ac6' cpuUtilization: idlePercent: 99.98 systemPercent: 0.01 userPercent: 0.01 cpus: 40 distribution: distribution: '"rhel"' version: "8.7" eventLogger: file hostname: anubis.hpc.rl.af.mil idMappings: gidmap: - container_id: 0 host_id: 2181 size: 1 - container_id: 1 host_id: 296608 size: 65536 uidmap: - container_id: 0 host_id: 2181 size: 1 - container_id: 1 host_id: 296608 size: 65536 kernel: 4.18.0-425.10.1.el8_7.x86_64 linkmode: dynamic logDriver: k8s-file memFree: 268191989760 memTotal: 270425149440 networkBackend: cni ociRuntime: name: runc package: runc-1.1.4-1.module+el8.7.0+17498+a7f63b89.x86_64 path: /usr/bin/runc version: |- runc version 1.1.4 spec: 1.0.2-dev go: go1.18.4 libseccomp: 2.5.2 os: linux remoteSocket: path: /run/user/2181/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.0-2.module+el8.7.0+17498+a7f63b89.x86_64 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.2 swapFree: 16001265664 swapTotal: 16001265664 uptime: 18h 20m 32.00s (Approximately 0.75 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.access.redhat.com - registry.redhat.io - docker.io store: configFile: /home/user/.config/containers/storage.conf containerStore: number: 1 paused: 0 running: 0 stopped: 1 graphDriverName: overlay graphOptions: {} graphRoot: /home/user/.local/share/containers/storage graphRootAllocated: 230741770240 graphRootUsed: 13806014464 graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 7 runRoot: /tmp/podman-run-2181/containers volumePath: /home/user/.local/share/containers/storage/volumes version: APIVersion: 4.2.0 Built: 1670845316 BuiltTime: Mon Dec 12 06:41:56 2022 GitCommit: "" GoVersion: go1.18.4 Os: linux OsArch: linux/amd64 Version: 4.2.0 ```


### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

bare metal host

NAME="Red Hat Enterprise Linux" VERSION="8.7 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"



### Additional information

Running containers as root seems to be working as expected.

containers / podman