search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.49k stars 2.31k forks source link

Cannot run container as root with Podman due to Cgroups issue with missing cpu.max sysfs file #13379

Closed pclass-sensonix closed 2 years ago

pclass-sensonix commented 2 years ago

/kind bug

Description

I can not run my container using Podman 4.0.0 on my Linux 5.4 i.MX8 device due to missing 'cpu.max' cgroups file. The system was built by Yocto. If I remove the cpus flag from my podman start command, the container starts properly. The cgroups configuration seems to be fine to me with the cpu cgroup controller shown as being enabled.

I also have a Microchip device with the same podman and systemd versions, again built by Yocto, and podman starts the same container image, with the failing set of arguments just fine. One difference is that this device runs Linux version 5.10.

The podman configuration files are the same for each device.

Steps to reproduce the issue:

  1. Boot Linux

  2. Run container with cpu limitation option: /usr/bin/podman run -it --root /container-storage/containers/storage --memory="60m" --memory-swap="60m" --cpus="0.75" company-base:1.4 /bin/sh

Describe the results you received:

Observed the following error:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# /usr/bin/podman run -it --root /container-storage/containers/storage --memory="60m"  --memory-swap="60m" --cpus="0.75"  company-base:1.4 /bin/sh
Error: crun: opening file `cpu.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected:

Podman container runs okay.

Additional information you deem important (e.g. issue happens only occasionally):

This happens every time when I specify the cpus flag. If I do not use the cpus flag, the container starts fine.

Output of podman version:

root@banner-fusion-imx8-gateway:/sys/fs/cgroup/libpod_parent# podman version
Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.16.5

Built:      Sat Feb 26 00:38:23 2022
OS/Arch:    linux/arm64

Output of podman info --debug:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# podman info
host:
  arch: arm64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 7e6de6678f6ed8a18661e1d5721b81ccee293b9b-dirty'
  cpus: 4
  distribution:
    distribution: unknown
    version: unknown
  eventLogger: journald
  hostname: company-device-imx8-gateway
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.24-iot-gate-imx8-2.5+gbabac008e5cf
  linkmode: dynamic
  logDriver: journald
  memFree: 3502698496
  memTotal: 4123029504
  networkBackend: cni
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2-dirty
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 0.4.1
      commit: unknown
  swapFree: 251654144
  swapTotal: 251654144
  uptime: 32m 30.4s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /container-storage/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /container-storage/downloaded
  imageStore:
    number: 6
  runRoot: /container-storage/containers/storage
  volumePath: /container-storage/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1645835903
  BuiltTime: Sat Feb 26 00:38:23 2022
  GitCommit: ""
  GoVersion: go1.16.5
  OsArch: linux/arm64
  Version: 4.0.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

This is a Yocto build

Additional info:

Cgroups info:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# ls -lrt /sys/fs/cgroup/libpod_parent/
total 0
drwxr-xr-x 2 root root 0 Feb 28 21:52 libpod-867d72b8630a1b0d505d25caef2251e3b8f127a15321ec907cc280c5b1aa6866
drwxr-xr-x 2 root root 0 Feb 28 21:52 libpod-1da15b6fe9e98594031601ad1ed7222f4eff96975afb4f4bc50d9591370575e1
drwxr-xr-x 2 root root 0 Feb 28 21:52 libpod-58c2293cbc3293965b9b26369f747d19c0accaf881e40b42f5bf025fae297b2b
-r--r--r-- 1 root root 0 Feb 28 21:55 cpuset.mems.effective
-rw-r--r-- 1 root root 0 Feb 28 21:55 cpuset.mems
-rw-r--r-- 1 root root 0 Feb 28 21:55 cpuset.cpus.partition
-r--r--r-- 1 root root 0 Feb 28 21:55 cpuset.cpus.effective
-rw-r--r-- 1 root root 0 Feb 28 21:55 cpuset.cpus
-rw-r--r-- 1 root root 0 Feb 28 21:55 cpu.weight.nice
-rw-r--r-- 1 root root 0 Feb 28 21:55 cpu.weight
-r--r--r-- 1 root root 0 Feb 28 21:55 cpu.stat
-rw-r--r-- 1 root root 0 Feb 28 21:55 cgroup.type
-rw-r--r-- 1 root root 0 Feb 28 21:55 cgroup.threads
-r--r--r-- 1 root root 0 Feb 28 21:55 cgroup.stat
-rw-r--r-- 1 root root 0 Feb 28 21:55 cgroup.procs
-rw-r--r-- 1 root root 0 Feb 28 21:55 cgroup.max.descendants
-rw-r--r-- 1 root root 0 Feb 28 21:55 cgroup.max.depth
-rw-r--r-- 1 root root 0 Feb 28 21:55 cgroup.freeze
-r--r--r-- 1 root root 0 Feb 28 21:55 cgroup.events
-r--r--r-- 1 root root 0 Feb 28 21:55 cgroup.controllers
drwxr-xr-x 2 root root 0 Feb 28 21:58 conmon
drwxr-xr-x 2 root root 0 Feb 28 21:58 libpod-732de01723fa5ab3111ebc4487e6897b35ace97a5bd2a0287b790186681bbb47
-rw-r--r-- 1 root root 0 Feb 28 21:58 cgroup.subtree_control
-rw-r--r-- 1 root root 0 Feb 28 22:20 pids.max
-r--r--r-- 1 root root 0 Feb 28 22:20 pids.events
-r--r--r-- 1 root root 0 Feb 28 22:20 pids.current
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.swap.max
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.swap.events
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.swap.current
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.stat
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.oom.group
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.min
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.max
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.low
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.high
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.events.local
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.events
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.current
-r--r--r-- 1 root root 0 Feb 28 22:20 io.stat
root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# ls -lrt /sys/fs/cgroup/libpod_parent/conmon 
total 0
-r--r--r-- 1 root root 0 Feb 28 21:58 cpuset.mems.effective
-r--r--r-- 1 root root 0 Feb 28 21:58 cpuset.cpus.effective
-rw-r--r-- 1 root root 0 Feb 28 21:58 cgroup.procs
-rw-r--r-- 1 root root 0 Feb 28 22:20 pids.max
-r--r--r-- 1 root root 0 Feb 28 22:20 pids.events
-r--r--r-- 1 root root 0 Feb 28 22:20 pids.current
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.swap.max
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.swap.events
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.swap.current
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.stat
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.oom.group
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.min
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.max
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.low
-rw-r--r-- 1 root root 0 Feb 28 22:20 memory.high
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.events.local
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.events
-r--r--r-- 1 root root 0 Feb 28 22:20 memory.current
-r--r--r-- 1 root root 0 Feb 28 22:20 io.stat
-rw-r--r-- 1 root root 0 Feb 28 22:20 cpuset.mems
-rw-r--r-- 1 root root 0 Feb 28 22:20 cpuset.cpus.partition
-rw-r--r-- 1 root root 0 Feb 28 22:20 cpuset.cpus
-rw-r--r-- 1 root root 0 Feb 28 22:20 cpu.weight.nice
-rw-r--r-- 1 root root 0 Feb 28 22:20 cpu.weight
-r--r--r-- 1 root root 0 Feb 28 22:20 cpu.stat
-rw-r--r-- 1 root root 0 Feb 28 22:20 cgroup.type
-rw-r--r-- 1 root root 0 Feb 28 22:20 cgroup.threads
-rw-r--r-- 1 root root 0 Feb 28 22:20 cgroup.subtree_control
-r--r--r-- 1 root root 0 Feb 28 22:20 cgroup.stat
-rw-r--r-- 1 root root 0 Feb 28 22:20 cgroup.max.descendants
-rw-r--r-- 1 root root 0 Feb 28 22:20 cgroup.max.depth
-rw-r--r-- 1 root root 0 Feb 28 22:20 cgroup.freeze
-r--r--r-- 1 root root 0 Feb 28 22:20 cgroup.events
-r--r--r-- 1 root root 0 Feb 28 22:20 cgroup.controllers

Cgroup Controllers:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent/conmon# cat /sys/fs/cgroup/user.slice/user-0.slice/user\@0.service/cgroup.controllers 
cpu io memory pids

Kernel Configuration:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent/conmon# !141
zcat /proc/config.gz | grep CGROUP
CONFIG_CGROUPS=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
# CONFIG_CGROUP_RDMA is not set
# CONFIG_CGROUP_FREEZER is not set
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_SOCK_CGROUP_DATA=y
# CONFIG_BLK_CGROUP_IOLATENCY is not set
# CONFIG_BLK_CGROUP_IOCOST is not set
# CONFIG_BFQ_CGROUP_DEBUG is not set
CONFIG_NETFILTER_XT_MATCH_CGROUP=m
# CONFIG_NET_CLS_CGROUP is not set
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y

Cgroup FS:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent/conmon# grep cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2

Linux Kernel Version:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent/conmon# uname -r
5.4.24-iot-gate-imx8-2.5+gbabac008e5cf

SystemD Version:

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent/conmon# systemctl --version
systemd 249 (249.7+)
+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK -SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid

Working Device Specifics

Working Device podman info:

root@company-device-gateway:/sys/fs/cgroup/libpod_parent# podman info
host:
  arch: arm
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 7e6de6678f6ed8a18661e1d5721b81ccee293b9b-dirty'
  cpus: 1
  distribution:
    distribution: unknown
    version: unknown
  eventLogger: journald
  hostname: company-device-gateway
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.50-linux4sam-2021.04
  linkmode: dynamic
  logDriver: journald
  memFree: 69480448
  memTotal: 253427712
  networkBackend: cni
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 0.4.1
      commit: unknown
  swapFree: 184020992
  swapTotal: 251654144
  uptime: 6h 21m 3.99s (Approximately 0.25 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 1
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /container-storage/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /container-storage/downloaded
  imageStore:
    number: 14
  runRoot: /container-storage/containers/storage
  volumePath: /container-storage/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1645491662
  BuiltTime: Tue Feb 22 01:01:02 2022
  GitCommit: ""
  GoVersion: go1.16.5
  OsArch: linux/arm
  Version: 4.0.0-dev

Working device Cgroups:

root@company-device-gateway:/sys/fs/cgroup/libpod_parent# ls -lrt /sys/fs/cgroup/libpod_parent/
total 0
drwxr-xr-x 2 root root 0 Nov 19 17:20 libpod-677ad7bf23dc287c00dc54ee6ea8b2779c38dffedd1b8b284054227fbee64133
-rw-r--r-- 1 root root 0 Feb 28 21:54 cgroup.subtree_control
drwxr-xr-x 2 root root 0 Feb 28 21:55 conmon
-rw-r--r-- 1 root root 0 Feb 28 22:21 pids.max
-r--r--r-- 1 root root 0 Feb 28 22:21 pids.events
-r--r--r-- 1 root root 0 Feb 28 22:21 pids.current
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.swap.max
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.swap.high
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.swap.events
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.swap.current
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.stat
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.oom.group
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.min
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.max
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.low
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.high
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.events.local
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.events
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.current
-rw-r--r-- 1 root root 0 Feb 28 22:21 cpu.weight.nice
-rw-r--r-- 1 root root 0 Feb 28 22:21 cpu.weight
-r--r--r-- 1 root root 0 Feb 28 22:21 cpu.stat
-rw-r--r-- 1 root root 0 Feb 28 22:21 cpu.max
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.type
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.threads
-r--r--r-- 1 root root 0 Feb 28 22:21 cgroup.stat
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.procs
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.max.descendants
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.max.depth
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.freeze
-r--r--r-- 1 root root 0 Feb 28 22:21 cgroup.events
-r--r--r-- 1 root root 0 Feb 28 22:21 cgroup.controllers
root@company-device-gateway:/sys/fs/cgroup/libpod_parent# ls -lrt /sys/fs/cgroup/libpod_parent/conmon/
total 0
-rw-r--r-- 1 root root 0 Feb 28 22:21 pids.max
-r--r--r-- 1 root root 0 Feb 28 22:21 pids.events
-r--r--r-- 1 root root 0 Feb 28 22:21 pids.current
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.swap.max
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.swap.high
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.swap.events
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.swap.current
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.stat
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.oom.group
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.min
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.max
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.low
-rw-r--r-- 1 root root 0 Feb 28 22:21 memory.high
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.events.local
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.events
-r--r--r-- 1 root root 0 Feb 28 22:21 memory.current
-rw-r--r-- 1 root root 0 Feb 28 22:21 cpu.weight.nice
-rw-r--r-- 1 root root 0 Feb 28 22:21 cpu.weight
-r--r--r-- 1 root root 0 Feb 28 22:21 cpu.stat
-rw-r--r-- 1 root root 0 Feb 28 22:21 cpu.max
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.type
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.threads
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.subtree_control
-r--r--r-- 1 root root 0 Feb 28 22:21 cgroup.stat
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.procs
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.max.descendants
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.max.depth
-rw-r--r-- 1 root root 0 Feb 28 22:21 cgroup.freeze
-r--r--r-- 1 root root 0 Feb 28 22:21 cgroup.events
-r--r--r-- 1 root root 0 Feb 28 22:21 cgroup.controllers

Working Device kernel config:

root@company-device-gateway:/sys/fs/cgroup/libpod_parent# zcat /proc/config.gz | grep CGROUP
CONFIG_CGROUPS=y
# CONFIG_BLK_CGROUP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
# CONFIG_CGROUP_RDMA is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_BPF=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_NETFILTER_XT_MATCH_CGROUP=m
# CONFIG_NET_CLS_CGROUP is not set
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y

Working Device Kernel / SystemD / and Cgroups fs:

systemctl --version
systemd 249 (249.7+)
+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK -SECCOMP -GCRYPT -GNUTLS -OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY +P11KIT -QRENCODE -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid

root@company-device-gateway:/sys/fs/cgroup/libpod_parent# uname -r
5.10.50-linux4sam-2021.04

root@company-device-gateway:/sys/fs/cgroup/libpod_parent# grep cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

baude commented 2 years ago

@giuseppe or @rhatdan ptal

giuseppe commented 2 years ago

looks like the cpu controller is not enabled.

What is the content of /sys/fs/cgroup/cgroup.subtree_control?

You may need to enable it manually running echo +cpu > /sys/fs/cgroup/cgroup.subtree_control as root

pclass-sensonix commented 2 years ago

@giuseppe - that's what I thought too, but when I look at the output of podman --info, it shows the cpu controller enabled:

cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids

I also tried your suggestion with the subtree_control file. The file already showed cpu, but I ran the command to enabled it again - however it still fails when I run the container with 'missing cpu.max':

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory pids
root@ company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# echo +cpu > /sys/fs/cgroup/cgroup.subtree_control
root@ company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory pids

Container fails with: Error: crun: opening file cpu.max for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

pclass-sensonix commented 2 years ago

I see other syses files related to cpu cgroup control, just not 'cpu.max'.

root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# ls *cpu*
cpu.stat  cpu.weight  cpu.weight.nice  cpuset.cpus  cpuset.cpus.effective  cpuset.cpus.partition  cpuset.mems  cpuset.mems.effective
giuseppe commented 2 years ago

it looks like the kernel was compiled without CONFIG_CFS_BANDWIDTH.

You either need to enable that in the kernel, or you cannot use --cpus

skepticoitusInteruptus commented 2 years ago

Hi @giuseppe :wave:

This comment came up in a search I did after I created containers/crun/issues/923 yesterday:

…

_What is the content of /sys/fs/cgroup/cgroup.subtree_control?_

_You may need to enable it manually running echo +cpu > /sys/fs/cgroup/cgroup.subtree_control as root_

I'm curious whether or not this issue might be related in any way to containers/crun/issues/923? Your thoughts, please?

TIA.

skepticoitusInteruptus commented 2 years ago

Hey @pclass-sensonix,

"root@company-device-imx8-gateway:/sys/fs/cgroup/libpod_parent# ls -lrt /sys/fs/cgroup/libpod_parent/"

I hope you don't mind my nosiness curiosity about your issue :nose:

I think you can you help me cement my understanding of cgroup v2 though. Please?

What does ls /sys/fs/cgroup show you? If there is a cpu.max file there, what does catting it show you?

I appreciate it's a long shot of a chance that you would still have a similar deployment after this amount of time.

TIA.