On GitHub Actions `ubuntu-22.04`, rootless podman now fails with `failed to write to /proc/self/oom_score_adj: Permission denied`

Issue Description

I've been using podman 4 on GitHub Actions ubuntu-22.04 machines to build RHEL images for ages.

It recently started to fail.

Steps to reproduce the issue

In your GitHub repository, add .github/workflows/build-test.yaml with
```
name: Build and test FreeIPA containers
```

on: push: workflow_dispatch:

jobs: build: name: Build image runs-on: ubuntu-22.04 timeout-minutes: 15 steps:

name: Install podman 4.* run: | sudo apt-get remove -y golang-github-containers-image \ && sudo mkdir -p /etc/apt/keyrings \ && curl -fsSL https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_$(lsb_release -rs)/Release.key \ | gpg --dearmor \ | sudo tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null \ && echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainersunstable.gpg] \ https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu$(lsb_release -rs)/ /" \ | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null \ && sudo apt-get update -y \ && sudo apt-get install -y podman skopeo
run: podman info
run: podman run --log-level=debug --rm registry.access.redhat.com/ubi9 cat /etc/redhat-release
```
2. Push and observe the GitHub actions started.
```

Describe the results you received

time="2023-12-18T07:19:33Z" level=info msg="podman filtering at log level debug"
time="2023-12-18T07:19:33Z" level=debug msg="Called run.PersistentPreRunE(podman run --log-level=debug --rm registry.access.redhat.com/ubi9 cat /etc/redhat-release)"
time="2023-12-18T07:19:33Z" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2023-12-18T07:19:33Z" level=debug msg="Initializing boltdb state at /home/runner/.local/share/containers/storage/libpod/bolt_state.db"
time="2023-12-18T07:19:33Z" level=debug msg="systemd-logind: Unknown object '/'."
time="2023-12-18T07:19:33Z" level=debug msg="Using graph driver overlay"
time="2023-12-18T07:19:33Z" level=debug msg="Using graph root /home/runner/.local/share/containers/storage"
time="2023-12-18T07:19:33Z" level=debug msg="Using run root /run/user/1001/containers"
time="2023-12-18T07:19:33Z" level=debug msg="Using static dir /home/runner/.local/share/containers/storage/libpod"
time="2023-12-18T07:19:33Z" level=debug msg="Using tmp dir /run/user/1001/libpod/tmp"
time="2023-12-18T07:19:33Z" level=debug msg="Using volume path /home/runner/.local/share/containers/storage/volumes"
time="2023-12-18T07:19:33Z" level=debug msg="Using transient store: false"
time="2023-12-18T07:19:33Z" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2023-12-18T07:19:33Z" level=debug msg="Cached value indicated that overlay is supported"
time="2023-12-18T07:19:33Z" level=debug msg="Cached value indicated that overlay is supported"
time="2023-12-18T07:19:33Z" level=debug msg="Cached value indicated that metacopy is not being used"
time="2023-12-18T07:19:33Z" level=debug msg="Cached value indicated that native-diff is usable"
time="2023-12-18T07:19:33Z" level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false"
time="2023-12-18T07:19:33Z" level=debug msg="Initializing event backend journald"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
time="2023-12-18T07:19:33Z" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2023-12-18T07:19:33Z" level=info msg="Setting parallel job count to 13"
time="2023-12-18T07:19:33Z" level=debug msg="Successfully loaded 1 networks"
time="2023-12-18T07:19:33Z" level=debug msg="Pulling image registry.access.redhat.com/ubi9 (policy: missing)"
time="2023-12-18T07:19:33Z" level=debug msg="Looking up image \"registry.access.redhat.com/ubi9\" in local containers storage"
time="2023-12-18T07:19:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2023-12-18T07:19:33Z" level=debug msg="Trying \"registry.access.redhat.com/ubi9:latest\" ..."
time="2023-12-18T07:19:33Z" level=debug msg="reference \"[overlay@/home/runner/.local/share/containers/storage+/run/user/1001/containers]registry.access.redhat.com/ubi9:latest\" does not resolve to an image ID"
time="2023-12-18T07:19:33Z" level=debug msg="Trying \"registry.access.redhat.com/ubi9:latest\" ..."
time="2023-12-18T07:19:33Z" level=debug msg="reference \"[overlay@/home/runner/.local/share/containers/storage+/run/user/1001/containers]registry.access.redhat.com/ubi9:latest\" does not resolve to an image ID"
time="2023-12-18T07:19:33Z" level=debug msg="Trying \"registry.access.redhat.com/ubi9\" ..."
time="2023-12-18T07:19:33Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2023-12-18T07:19:33Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
time="2023-12-18T07:19:33Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/shortnames.conf\""
time="2023-12-18T07:19:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2023-12-18T07:19:33Z" level=debug msg="Attempting to pull candidate registry.access.redhat.com/ubi9:latest for registry.access.redhat.com/ubi9"
time="2023-12-18T07:19:33Z" level=debug msg="parsed reference into \"[overlay@/home/runner/.local/share/containers/storage+/run/user/1001/containers]registry.access.redhat.com/ubi9:latest\""
Trying to pull registry.access.redhat.com/ubi9:latest...
time="2023-12-18T07:19:33Z" level=debug msg="Copying source image //registry.access.redhat.com/ubi9:latest to destination image [overlay@/home/runner/.local/share/containers/storage+/run/user/1001/containers]registry.access.redhat.com/ubi9:latest"
time="2023-12-18T07:19:33Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2023-12-18T07:19:38Z" level=debug msg="running conmon: /usr/bin/conmon" args="[--api-version 1 -c 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88 -u 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88 -r /usr/bin/crun -b /home/runner/.local/share/containers/storage/overlay-containers/052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88/userdata -p /run/user/1001/containers/overlay-containers/052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88/userdata/pidfile -n relaxed_buck --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog --conmon-pidfile /run/user/1001/containers/overlay-containers/052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/runner/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/runner/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

time="2023-12-18T07:19:38Z" level=debug msg="Cleaning up container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88"
time="2023-12-18T07:19:38Z" level=debug msg="Tearing down network namespace at /run/user/1001/netns/netns-62ca624a-b9b8-deed-7c4e-1532098d2b03 for container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88"
time="2023-12-18T07:19:38Z" level=debug msg="Unmounted container \"052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88\""
time="2023-12-18T07:19:38Z" level=debug msg="Removing container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88"
time="2023-12-18T07:19:38Z" level=debug msg="Cleaning up container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88"
time="2023-12-18T07:19:38Z" level=debug msg="Network is already cleaned up, skipping..."
time="2023-12-18T07:19:38Z" level=debug msg="Container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88 storage is already unmounted, skipping..."
time="2023-12-18T07:19:38Z" level=debug msg="Removing all exec sessions for container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88"
time="2023-12-18T07:19:38Z" level=debug msg="Container 052a08a0a43490878039595a757d51bc89987bd3519ba76e22b749447afc9c88 storage is already unmounted, skipping..."
time="2023-12-18T07:19:38Z" level=debug msg="ExitCode msg: \"container create failed (no logs from conmon): conmon bytes \\\"\\\": readobjectstart: expect { or n, but found \\x00, error found in #0 byte of ...||..., bigger context ...||...\""
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
time="2023-12-18T07:19:38Z" level=debug msg="Shutting down engines"

Describe the results you expected

Container created and run

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.9-0ubuntu22.04+obs17.1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.9, commit: '
  cpuUtilization:
    idlePercent: 86.88
    systemPercent: 5.69
    userPercent: 7.43
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2048
  hostname: fv-az1040-680
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 127
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.2.0-1018-azure
  linkmode: dynamic
  logDriver: journald
  memFree: 14219862016
  memTotal: 16757788672
  networkBackend: netavark
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/runner/.local/share/containers/storage
  graphRootAllocated: 89297309696
  graphRootUsed: 61875249152
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

GitHub ubuntu-22.04 and podman from https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_$(lsb_release -rs)/

Additional information

Deterministic.

I'm aware of https://github.com/containers/podman/issues/3024. However, that has been closed since 2019, people recently seem piling reports of hitting the issue again, and here I'd like to specifically focus on getting podman 4 working on GitHub Actions again. Ideally things should be packaged in such a way that no special tweaks are needed, as was the situation until last week.

Speaking about last time when this worked, in https://github.com/freeipa/freeipa-container/actions/runs/7217969905/job/19669690006 I see

Preparing to unpack .../00-aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64.deb ...
Unpacking aardvark-dns (1.6.0-0ubuntu22.04+obs34.25) ...
Preparing to unpack .../01-conmon_2%3a2.1.8-0ubuntu22.04+obs16.39_amd64.deb ...
Unpacking conmon (2:2.1.8-0ubuntu22.04+obs16.39) over (2.0.25+ds1-1.1) ...
Selecting previously unselected package libprotobuf23:amd64.
Preparing to unpack .../02-libprotobuf23_3.12.4-1ubuntu7.22.04.1_amd64.deb ...
Unpacking libprotobuf23:amd64 (3.12.4-1ubuntu7.22.04.1) ...
Selecting previously unselected package python3-protobuf.
Preparing to unpack .../03-python3-protobuf_3.12.4-1ubuntu7.22.04.1_amd64.deb ...
Unpacking python3-protobuf (3.12.4-1ubuntu7.22.04.1) ...
Selecting previously unselected package libnet1:amd64.
Preparing to unpack .../04-libnet1_1.1.6+dfsg-3.1build3_amd64.deb ...
Unpacking libnet1:amd64 (1.1.6+dfsg-3.1build3) ...
Selecting previously unselected package libprotobuf-c1:amd64.
Preparing to unpack .../05-libprotobuf-c1_1.3.3-1ubuntu2.1_amd64.deb ...
Unpacking libprotobuf-c1:amd64 (1.3.3-1ubuntu2.1) ...
Selecting previously unselected package criu.
Preparing to unpack .../06-criu_3.16.1-2_amd64.deb ...
Unpacking criu (3.16.1-2) ...
Preparing to unpack .../07-crun_101%3a1.12-0ubuntu22.04+obs65.13_amd64.deb ...
Unpacking crun (101:1.12-0ubuntu22.04+obs65.13) over (0.17+dfsg-1.1) ...
Selecting previously unselected package netavark.
Preparing to unpack .../08-netavark_1.3.0-0ubuntu22.04+obs22.9_amd64.deb ...
Unpacking netavark (1.3.0-0ubuntu22.04+obs22.9) ...
Selecting previously unselected package containers-common.
Preparing to unpack .../09-containers-common_4%3a1-0ubuntu22.04+obs20.56_amd64.deb ...
Unpacking containers-common (4:1-0ubuntu22.04+obs20.56) ...
Selecting previously unselected package podman.
Preparing to unpack .../10-podman_4%3a4.6.2-0ubuntu22.04+obs81.12_amd64.deb ...
Unpacking podman (4:4.6.2-0ubuntu22.04+obs81.12) ...
dpkg: warning: unable to delete old directory '/etc/cni/net.d': Directory not empty
dpkg: warning: unable to delete old directory '/etc/cni': Directory not empty
Selecting previously unselected package podman-gvproxy.
Preparing to unpack .../11-podman-gvproxy_4%3a4.6.2-0ubuntu22.04+obs81.12_amd64.deb ...
Unpacking podman-gvproxy (4:4.6.2-0ubuntu22.04+obs81.12) ...
Selecting previously unselected package skopeo.
Preparing to unpack .../12-skopeo_2%3a1.13.3-0ubuntu22.04+obs31.29_amd64.deb ...
Unpacking skopeo (2:1.13.3-0ubuntu22.04+obs31.29) ...
Setting up libprotobuf23:amd64 (3.12.4-1ubuntu7.22.04.1) ...
Setting up libnet1:amd64 (1.1.6+dfsg-3.1build3) ...
Setting up libprotobuf-c1:amd64 (1.3.3-1ubuntu2.1) ...
Setting up conmon (2:2.1.8-0ubuntu22.04+obs16.39) ...
Setting up podman-gvproxy (4:4.6.2-0ubuntu22.04+obs81.12) ...
Setting up aardvark-dns (1.6.0-0ubuntu22.04+obs34.25) ...
Setting up python3-protobuf (3.12.4-1ubuntu7.22.04.1) ...
Setting up netavark (1.3.0-0ubuntu22.04+obs22.9) ...
Setting up criu (3.16.1-2) ...
Setting up crun (101:1.12-0ubuntu22.04+obs65.13) ...
Setting up containers-common (4:1-0ubuntu22.04+obs20.56) ...
Installing new version of config file /etc/containers/policy.json ...

Configuration file '/etc/containers/registries.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.
Setting up skopeo (2:1.13.3-0ubuntu22.04+obs31.29) ...
Setting up podman (4:4.6.2-0ubuntu22.04+obs81.12) ...
Processing triggers for libc-bin (2.35-0ubuntu3.5) ...
Processing triggers for man-db (2.10.2-1) ...

This week the failed job https://github.com/freeipa/freeipa-container/actions/runs/7243634318/job/19733472131 showed

Preparing to unpack .../00-aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64.deb ...
Unpacking aardvark-dns (1.6.0-0ubuntu22.04+obs34.25) ...
Preparing to unpack .../01-conmon_2%3a2.1.9-0ubuntu22.04+obs17.1_amd64.deb ...
Unpacking conmon (2:2.1.9-0ubuntu22.04+obs17.1) over (2.0.25+ds1-1.1) ...
Selecting previously unselected package libprotobuf23:amd64.
Preparing to unpack .../02-libprotobuf23_3.12.4-1ubuntu7.22.04.1_amd64.deb ...
Unpacking libprotobuf23:amd64 (3.12.4-1ubuntu7.22.04.1) ...
Selecting previously unselected package python3-protobuf.
Preparing to unpack .../03-python3-protobuf_3.12.4-1ubuntu7.22.04.1_amd64.deb ...
Unpacking python3-protobuf (3.12.4-1ubuntu7.22.04.1) ...
Selecting previously unselected package libnet1:amd64.
Preparing to unpack .../04-libnet1_1.1.6+dfsg-3.1build3_amd64.deb ...
Unpacking libnet1:amd64 (1.1.6+dfsg-3.1build3) ...
Selecting previously unselected package libprotobuf-c1:amd64.
Preparing to unpack .../05-libprotobuf-c1_1.3.3-1ubuntu2.1_amd64.deb ...
Unpacking libprotobuf-c1:amd64 (1.3.3-1ubuntu2.1) ...
Selecting previously unselected package criu.
Preparing to unpack .../06-criu_3.16.1-2_amd64.deb ...
Unpacking criu (3.16.1-2) ...
Preparing to unpack .../07-crun_101%3a1.12-0ubuntu22.04+obs65.13_amd64.deb ...
Unpacking crun (101:1.12-0ubuntu22.04+obs65.13) over (0.17+dfsg-1.1) ...
Selecting previously unselected package netavark.
Preparing to unpack .../08-netavark_1.3.0-0ubuntu22.04+obs22.9_amd64.deb ...
Unpacking netavark (1.3.0-0ubuntu22.04+obs22.9) ...
Selecting previously unselected package containers-common.
Preparing to unpack .../09-containers-common_4%3a1-0ubuntu22.04+obs20.56_amd64.deb ...
Unpacking containers-common (4:1-0ubuntu22.04+obs20.56) ...
Selecting previously unselected package podman.
Preparing to unpack .../10-podman_4%3a4.6.2-0ubuntu22.04+obs81.12_amd64.deb ...
Unpacking podman (4:4.6.2-0ubuntu22.04+obs81.12) ...
dpkg: warning: unable to delete old directory '/etc/cni/net.d': Directory not empty
dpkg: warning: unable to delete old directory '/etc/cni': Directory not empty
Selecting previously unselected package podman-gvproxy.
Preparing to unpack .../11-podman-gvproxy_4%3a4.6.2-0ubuntu22.04+obs81.12_amd64.deb ...
Unpacking podman-gvproxy (4:4.6.2-0ubuntu22.04+obs81.12) ...
Selecting previously unselected package skopeo.
Preparing to unpack .../12-skopeo_2%3a1.13.3-0ubuntu22.04+obs31.29_amd64.deb ...
Unpacking skopeo (2:1.13.3-0ubuntu22.04+obs31.29) ...
Setting up libprotobuf23:amd64 (3.12.4-1ubuntu7.22.04.1) ...
Setting up libnet1:amd64 (1.1.6+dfsg-3.1build3) ...
Setting up libprotobuf-c1:amd64 (1.3.3-1ubuntu2.1) ...
Setting up conmon (2:2.1.9-0ubuntu22.04+obs17.1) ...
Setting up podman-gvproxy (4:4.6.2-0ubuntu22.04+obs81.12) ...
Setting up aardvark-dns (1.6.0-0ubuntu22.04+obs34.25) ...
Setting up python3-protobuf (3.12.4-1ubuntu7.22.04.1) ...
Setting up netavark (1.3.0-0ubuntu22.04+obs22.9) ...
Setting up criu (3.16.1-2) ...
Setting up crun (101:1.12-0ubuntu22.04+obs65.13) ...
Setting up containers-common (4:1-0ubuntu22.04+obs20.56) ...
Installing new version of config file /etc/containers/policy.json ...

Configuration file '/etc/containers/registries.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.
Setting up skopeo (2:1.13.3-0ubuntu22.04+obs31.29) ...
Setting up podman (4:4.6.2-0ubuntu22.04+obs81.12) ...
Processing triggers for libc-bin (2.35-0ubuntu3.5) ...
Processing triggers for man-db (2.10.2-1) ...

containers / podman