Podman complains about cgroups v1 when cgroups-v2 is supported on the host

vwbusguy commented 2 months ago

Issue Description

I'm seeing this when running podman on Kubernetes rootless via privileged mode, but also able to replicate it directly with podman on the kubernetes host.

When trying to do some CI automations with podman, we get numerous log entries for:

Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.

Steps to reproduce the issue

Create a podman pod (or otherwise run podman on a host with cgroups2)
Do normal podman things (build, run, etc.)
Observe the continuous log messages

Describe the results you received

[user@ucsb-pstat-github-comm-594ct-container-image-main-9-c8p76-k1r24 agent]$ grep cgroup /proc/filesystems 
nodev   cgroup
nodev   cgroup2
[user@ucsb-pstat-github-comm-594ct-container-image-main-9-c8p76-k1r24 agent]$ podman info
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '

Describe the results you expected

These warnings should not be given in environments that support cgroups-v2.

podman info output

From within a pod running in RKE2 on the quay.io/containers/aio:latest image:

user@ucsb-pstat-github-comm-594ct-container-image-main-9-c8p76-k1r24 agent]$ podman info
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.6
    systemPercent: 0.13
    userPercent: 0.27
  cpus: 256
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: file
  freeLocks: 2048
  hostname: ucsb-pstat-github-comm-594ct-container-image-main-9-c8p76-k1r24
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 5.14.21-150500.55.44-default
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 330199961600
  memTotal: 540754264064
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /tmp/storage-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240821.g1d6142f-1.fc40.x86_64
    version: |
      pasta 0^20240821.g1d6142f-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/storage-run-1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 3197h 47m 3.00s (Approximately 133.21 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 773641093120
  graphRootUsed: 572532244480
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /tmp/storage-run-1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Wed Aug 21 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

For now, I've added the PODMAN_IGNORE_CGROUPSV1_WARNING environment variable in my CI, but this message seems to be constantly giving an unnecessary call to action.

I have also tried mounting /sys/fs/cgroup/ from the host as a much earlier GitHub issue commented suggested, but it did not have an effect.

Additional information

I would assume that a grep cgroup2 /proc/filesystems check before showing this message might be sufficient, assuming there aren't any other side effects, which I've yet to discover.

giuseppe commented 2 months ago

can you please show the output of cat /proc/self/mountinfo?

I think you are seeing a cgroup2 mount, but you are using the hybrid mount model (that is cgroupv2 mounted under a cgroupv1 hierarchy)

vwbusguy commented 2 months ago

Because this is a k8s host (Rancher Elemental SLE Micro), there's a ton of overlay output there, so I did it with an cgroup grep:

grep cgroup /proc/self/mountinfo
30 24 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:26 - tmpfs tmpfs ro,size=4096k,nr_inodes=1024,mode=755,inode64
31 30 0:27 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:27 - cgroup2 cgroup2 rw
32 30 0:28 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:28 - cgroup cgroup rw,xattr,name=systemd
36 30 0:32 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:29 - cgroup cgroup rw,pids
37 30 0:33 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:30 - cgroup cgroup rw,cpu,cpuacct
38 30 0:34 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:31 - cgroup cgroup rw,net_cls,net_prio
39 30 0:35 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:32 - cgroup cgroup rw,rdma
40 30 0:36 / /sys/fs/cgroup/misc rw,nosuid,nodev,noexec,relatime shared:33 - cgroup cgroup rw,misc
41 30 0:37 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:34 - cgroup cgroup rw,blkio
42 30 0:38 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:35 - cgroup cgroup rw,memory
43 30 0:39 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:36 - cgroup cgroup rw,cpuset
44 30 0:40 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:37 - cgroup cgroup rw,hugetlb
45 30 0:41 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:38 - cgroup cgroup rw,freezer
46 30 0:42 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:39 - cgroup cgroup rw,perf_event
47 30 0:43 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:40 - cgroup cgroup rw,devices

Does podman not support cgroupsv2 in a unified hierarchy? I thought it did.

vwbusguy commented 2 months ago

Ah, so if cgroups2 is mounted in /unified, it's not actually a unified hierarchy but a hybrid one. That's a little confusing.

https://github.com/containers/podman/issues/4659#issuecomment-563378217

It seems I need to followup with SUSE support on the ramifications of switching the hierarchy on these nodes. The message could be more clear from podman's end though, as it is confusing to have cgroups2 enabled and still see this message because it's not enabled a specific way that podman supports.

giuseppe commented 2 months ago

as it is confusing to have cgroups2 enabled and still see this message because it's not enabled a specific way that podman supports.

it is not really a podman limitation, but more of a kernel+systemd thing. If a controller is enabled (like memory, or cpu) on cgroup v1 then it cannot be used on cgroup v2. IMO "hybrid mode" was good only to experiment with cgroup v2 but it is not really usable as it requires manual changes to make it work. You'd need to make sure controllers are configured for cgroup v2 and not cgroup v1 at startup, so for podman&crun we decided to not support it

containers / podman