search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.34k stars 2.38k forks source link

Ubuntu Jammy kernel is hogging memory #18761

Closed shieldwed closed 1 year ago

shieldwed commented 1 year ago

Issue Description

Restarting Podman containers or running containers with health checks leads to raising memory usage until the system gets stuck (after 4.4 hours, SystemD started to terminate services due to watchdog timeouts). On a regular system, with about 20 containers running it happens over about 2 weeks, but it can be condensed down to hours (see steps below).

Collecting metrics on all processes, I noticed that all of them were using more or less the same amount of memory, so user space processes didn't seem to claim more memory over time. However, the /proc/meminfo contains the metric Percpu which was increasing over time just about the same amount of available memory decreased.

Searching the web I found https://access.redhat.com/solutions/6740861 (undisclosed information) and https://bugzilla.redhat.com/show_bug.cgi?id=2004037 (older kernel) which didn't help me particularly.

Steps to reproduce the issue

Steps to reproduce the issue:

  1. Run 
    for i in $(seq 100);do podman run --detach --name=test$i --replace --health-cmd="/bin/true" --health-interval=10s --health-timeout=10s busybox /bin/sleep 86400; done
  2. Watch the output of the following command rise until the system becomes unresponsive: 
    while :; do grep Percpu /proc/meminfo; sleep 180; done

Describe the results you received

Before starting, any podman containers:

# free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.7Gi       989Mi        37Mi       1.1Gi       1.8Gi
Swap:          3.0Gi          0B       3.0Gi
# grep Percpu /proc/meminfo
Percpu:           166080 kB

Once all containers are started:

# free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.6Gi       124Mi        44Mi       1.1Gi       904Mi
Swap:          3.0Gi       1.0Mi       3.0Gi
# grep Percpu /proc/meminfo
Percpu:           644160 kB
Now watch the `Percpu` metric rising: ```bash # while :; do grep Percpu /proc/meminfo; sleep 180; done Percpu: 736320 kB Percpu: 815040 kB Percpu: 840960 kB Percpu: 861120 kB Percpu: 901440 kB Percpu: 910080 kB Percpu: 989760 kB Percpu: 1076160 kB Percpu: 1086720 kB Percpu: 1120320 kB Percpu: 1141440 kB Percpu: 1228800 kB Percpu: 1214400 kB Percpu: 1335360 kB Percpu: 1289280 kB Percpu: 1272960 kB Percpu: 1332480 kB Percpu: 1394880 kB Percpu: 1410240 kB Percpu: 1416000 kB Percpu: 1458240 kB Percpu: 1487040 kB Percpu: 1510080 kB Percpu: 1521600 kB Percpu: 1534080 kB Percpu: 1536000 kB Percpu: 1605120 kB Percpu: 1618560 kB Percpu: 1659840 kB Percpu: 1657920 kB Percpu: 1700160 kB Percpu: 1746240 kB Percpu: 1776960 kB Percpu: 1794240 kB Percpu: 1785600 kB Percpu: 1794240 kB Percpu: 1819200 kB Percpu: 1830720 kB Percpu: 1870080 kB Percpu: 1923840 kB Percpu: 1933440 kB Percpu: 1883520 kB Percpu: 1882560 kB Percpu: 1939200 kB Percpu: 1896000 kB Percpu: 1959360 kB Percpu: 2005440 kB Percpu: 1954560 kB Percpu: 1954560 kB Percpu: 2066880 kB Percpu: 2168640 kB Percpu: 2145600 kB Percpu: 2209920 kB Percpu: 2233920 kB Percpu: 2272320 kB Percpu: 2233920 kB Percpu: 2272320 kB Percpu: 2309760 kB Percpu: 2308800 kB Percpu: 2339520 kB Percpu: 2385600 kB Percpu: 2402880 kB Percpu: 2402880 kB Percpu: 2402880 kB Percpu: 2441280 kB Percpu: 2462400 kB Percpu: 2462400 kB Percpu: 2462400 kB Percpu: 2462400 kB Percpu: 2462400 kB Percpu: 2462400 kB Percpu: 2446080 kB Percpu: 2445120 kB Percpu: 2441280 kB Percpu: 2441280 kB Percpu: 2441280 kB Percpu: 2439360 kB Percpu: 2439360 kB Percpu: 2439360 kB Percpu: 2439360 kB Percpu: 2437440 kB Percpu: 2437440 kB Percpu: 2437440 kB Percpu: 2437440 kB Percpu: 2437440 kB Percpu: 2437440 kB Percpu: 2437440 kB Percpu: 2437440 kB ```

Describe the results you expected

Memory usage should not raise over time merely by employing health checks or restarting containers.

podman info output

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: testmachine
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 33
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 33
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-72-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 135667712
  memTotal: 4041187328
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux 
  remoteSocket:
    exists: true
    path: /run/user/33/podman/podman.sock
  security: 
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |- 
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 3218567168
  swapTotal: 3221221376
  uptime: 20m 19.22s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/www-data/.config/containers/storage.conf
  containerStore:
    number: 100
    paused: 0
    running: 100
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/www-data/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/33/containers
  volumePath: /home/www-data/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

# uname -a
Linux testmachine 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
# apt show podman
Package: podman
Version: 3.4.4+ds1-1ubuntu1
Built-Using: containerd (= 1.5.9-0ubuntu1), docker-registry (= 2.7.1+ds2-7build2), docker.io (= 20.10.12-0ubuntu2), golang-1.17 (= 1.17.3-1ubuntu1), golang-dbus (= 5.0.3-2), golang-fsnotify (= 1.5.1-1), golang-ginkgo (= 1.14.2-1), golang-github-acarl005-stripansi (= 0.0~git20180116.5a71ef0-2), golang-github-appc-cni (= 0.8.1-1), golang-github-beorn7-perks (= 1.0.1-1), golang-github-blang-semver (= 3.6.1-2), golang-github-buger-goterm (= 0.0+git20181115.c206103-2), golang-github-cespare-xxhash (= 2.1.1-2), golang-github-checkpoint-restore-go-criu (= 5.1.0-1), golang-github-chzyer-readline (= 1.4.39.g2972be2-3), golang-github-containernetworking-plugins (= 0.9.1+ds1-1), golang-github-containers-buildah (= 1.23.1+ds1-2), golang-github-containers-common (= 0.44.4+ds1-1), golang-github-containers-image (= 5.16.0-3), golang-github-containers-ocicrypt (= 1.0.3-1), golang-github-containers-psgo (= 1.5.2-1), golang-github-containers-storage (= 1.36.0+ds1-2), golang-github-coreos-bbolt (= 1.3.5-1), golang-github-coreos-go-iptables (= 0.5.0-1), golang-github-coreos-go-systemd (= 22.3.2-1), golang-github-cyphar-filepath-securejoin (= 0.2.2-2), golang-github-davecgh-go-spew (= 1.1.1-2), golang-github-disiqueira-gotree (= 3.0.2-2), golang-github-docker-docker-credential-helpers (= 0.6.4+ds1-1), golang-github-docker-go-connections (= 0.4.0-3), golang-github-docker-go-metrics (= 0.0.1-1), golang-github-docker-go-units (= 0.4.0-3), golang-github-docker-libtrust (= 0.0~git20150526.0.9cbd2a1-3.1), golang-github-dtylman-scp (= 0.0~git20181017.f3000a3-3), golang-github-fsouza-go-dockerclient (= 1.6.6-1ubuntu1), golang-github-fullsailor-pkcs7 (= 0.0~git20210826.33d0574-1), golang-github-ghodss-yaml (= 1.0.0-1.1), golang-github-go-logr-logr (= 0.4.0-1), golang-github-google-go-cmp (= 0.5.6-1), golang-github-google-go-intervals (= 0.0.2-2), golang-github-google-gofuzz (= 1.2.0-1), golang-github-google-shlex (= 0.0~git20191202.e7afc7f-1), golang-github-google-uuid (= 1.2.0-1), golang-github-gorilla-handlers (= 1.4.2-1), golang-github-gorilla-mux (= 1.8.0-1), golang-github-hashicorp-errwrap (= 1.1.0-1), golang-github-hashicorp-go-multierror (= 1.1.0-1), golang-github-influxdata-tail (= 1.0.0+git20180327.c434825-4), golang-github-ishidawataru-sctp (= 0.0+git20210707.9a39160-1), golang-github-jinzhu-copier (= 0.3.2-2), golang-github-json-iterator-go (= 1.1.10-2), golang-github-juju-ansiterm (= 0.0~git20210929.8b71cc9-1), golang-github-klauspost-compress (= 1.13.6+ds1-2), golang-github-klauspost-pgzip (= 1.2.5-1), golang-github-lunixbochs-vtclean (= 1.0.0-1), golang-github-manifoldco-promptui (= 0.8.0-2), golang-github-mattn-go-colorable (= 0.1.12-1), golang-github-mattn-go-isatty (= 0.0.14-1), golang-github-mattn-go-runewidth (= 0.0.13-2), golang-github-mitchellh-mapstructure (= 1.4.3-1), golang-github-moby-sys (= 0.0~git20201113.5a29239-1), golang-github-moby-term (= 0.0~git20210619.3f7ff69-1), golang-github-modern-go-concurrent (= 1.0.3-1.1), golang-github-modern-go-reflect2 (= 1.0.1-1), golang-github-opencontainers-go-digest (= 1.0.0-1), golang-github-opencontainers-image-spec (= 1.0.2-2), golang-github-opencontainers-runtime-tools (= 0.9.0+dfsg-2), golang-github-opencontainers-selinux (= 1.8.5-1), golang-github-opencontainers-specs (= 1.0.2.66.g20a2d97-1), golang-github-openshift-imagebuilder (= 1.2.1+ds1-3), golang-github-pkg-errors (= 0.9.1-2), golang-github-pmezard-go-difflib (= 1.0.0-3), golang-github-proglottis-gpgme (= 0.1.1-1), golang-github-prometheus-client-golang (= 1.11.0-3), golang-github-prometheus-client-model (= 0.2.0-3), golang-github-prometheus-common (= 0.32.1-5), golang-github-prometheus-procfs (= 0.7.3-1), golang-github-rivo-uniseg (= 0.2.0-1), golang-github-safchain-ethtool (= 0.1.0-1), golang-github-spf13-cobra (= 1.3.0-1), golang-github-spf13-pflag (= 1.0.5-2), golang-github-ulikunitz-xz (= 0.5.6-2), golang-github-vbatts-tar-split (= 0.11.1-2), golang-github-vbauerster-mpb (= 7.0.3-1), golang-github-vishvananda-netlink (= 1.1.0-2), golang-github-vishvananda-netns (= 0.0~git20200728.db3c7e5-1), golang-github-vividcortex-ewma (= 1.1.1-2), golang-github-xeipuuv-gojsonpointer (= 0.0~git20190905.02993c4-2), golang-github-xeipuuv-gojsonreference (= 0.0~git20180127.bd5ef7b-2), golang-github-xeipuuv-gojsonschema (= 1.2.0-2), golang-go-patricia (= 2.2.6-5), golang-go-zfs (= 2.1.1.49.gf784269-1), golang-go.crypto (= 1:0.0~git20211202.5770296-1), golang-gocapability-dev (= 0.0+git20200815.42c35b4-1), golang-gogoprotobuf (= 1.3.2-1), golang-golang-x-net (= 1:0.0+git20211209.491a49a+dfsg-1), golang-golang-x-sync (= 0.0~git20210220.036812b-1), golang-golang-x-sys (= 0.0~git20211216.1d35b9e-1), golang-golang-x-term (= 0.0~git20210615.6886f2d-1), golang-golang-x-text (= 0.3.7-1), golang-golang-x-xerrors (= 0.0~git20200804.5ec99f8-1), golang-gomega (= 1.10.3-1), golang-google-genproto (= 0.0~git20200413.b5235f6-1), golang-google-grpc (= 1.29.1-0ubuntu1), golang-google-protobuf (= 1.27.1-1), golang-gopkg-inf.v0 (= 0.9.1-1), golang-gopkg-square-go-jose.v2 (= 2.6.0-1), golang-gopkg-tomb.v1 (= 0.0~git20141024.0.dd63297-7), golang-gopkg-yaml.v3 (= 3.0.0~git20210107.496545a-1), golang-goprotobuf (= 1.3.4-2), golang-k8s-klog (= 2.5.0-2), golang-k8s-sigs-structured-merge-diff (= 4.1.2+ds1-1), golang-logrus (= 1.7.0-2), golang-protobuf-extensions (= 1.0.1-1), golang-toml (= 0.4.1+git20210923.e0af6a2-1), golang-yaml.v2 (= 2.4.0-1), rootlesskit (= 0.14.2-1build1), runc (= 1.0.3-0ubuntu1)
Priority: optional
Section: universe/admin
Source: libpod
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Go Packaging Team <pkg-go-maintainers@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 38.1 MB
Depends: init-system-helpers (>= 1.52), libc6 (>= 2.34), libdevmapper1.02.1 (>= 2:1.02.97), libgpgme11 (>= 1.4.1), libseccomp2 (>= 2.4.1), conmon (>= 2.0.18~), containernetworking-plugins (>= 0.8.7), golang-github-containers-common, crun | runc (>= 1.0.0~rc92~)
Recommends: buildah (>= 1.21.0), fuse-overlayfs (>= 1.0.0~), slirp4netns (>= 0.4.1~), catatonit | tini | dumb-init, uidmap, golang-github-containernetworking-plugin-dnsname
Suggests: containers-storage, docker-compose, iptables
Breaks: buildah (<< 1.10.1-6), fuse-overlayfs (<< 0.7.1), slirp4netns (<< 0.4.1)
Homepage: https://github.com/containers/podman
Download-Size: 10.4 MB
APT-Manual-Installed: yes
APT-Sources: https://mirror.nine.ch/ubuntu jammy/universe amd64 Packages
Description: engine to run OCI-based containers in Pods

Additional information

Kernel stack trace while CPU was working with low free memory: ``` [15646.322414] sysrq: Show backtrace of all active CPUs [15646.323161] NMI backtrace for cpu 1 [15646.323165] CPU: 1 PID: 3383669 Comm: bash Not tainted 5.15.0-72-generic #79-Ubuntu [15646.323168] Hardware name: Nutanix AHV, BIOS 0.0.0 02/06/2015 [15646.323173] Call Trace: [15646.323175] [15646.323184] show_stack+0x52/0x5c [15646.323191] dump_stack_lvl+0x4a/0x63 [15646.323196] dump_stack+0x10/0x16 [15646.323197] nmi_cpu_backtrace.cold+0x4d/0x93 [15646.323199] ? lapic_can_unplug_cpu+0x90/0x90 [15646.323205] nmi_trigger_cpumask_backtrace+0xec/0x100 [15646.323211] arch_trigger_cpumask_backtrace+0x19/0x20 [15646.323214] sysrq_handle_showallcpus+0x17/0x20 [15646.323217] __handle_sysrq.cold+0xcc/0x1a6 [15646.323222] ? apparmor_file_permission+0x70/0x160 [15646.323226] write_sysrq_trigger+0x28/0x40 [15646.323228] proc_reg_write+0x5e/0xa0 [15646.323232] ? __cond_resched+0x1a/0x50 [15646.323237] vfs_write+0xc7/0x270 [15646.323240] ksys_write+0x67/0xf0 [15646.323242] __x64_sys_write+0x19/0x20 [15646.323244] do_syscall_64+0x5c/0xc0 [15646.323249] ? irqentry_exit+0x1d/0x30 [15646.323251] ? exc_page_fault+0x89/0x170 [15646.323253] entry_SYSCALL_64_after_hwframe+0x61/0xcb [15646.323257] RIP: 0033:0x7fce27f19a37 [15646.323265] Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [15646.323267] RSP: 002b:00007ffe748e8638 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [15646.323269] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fce27f19a37 [15646.323270] RDX: 0000000000000002 RSI: 00005612fbff7d10 RDI: 0000000000000001 [15646.323271] RBP: 00005612fbff7d10 R08: 00007fce27fd6460 R09: 000000007fffffff [15646.323272] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002 [15646.323273] R13: 00007fce2801f780 R14: 00007fce2801b600 R15: 00007fce2801aa00 [15646.323280] [15646.323285] Sending NMI from CPU 1 to CPUs 0,2-3: [15646.323305] NMI backtrace for cpu 0 [15646.323308] CPU: 0 PID: 107 Comm: kswapd0 Not tainted 5.15.0-72-generic #79-Ubuntu [15646.323310] Hardware name: Nutanix AHV, BIOS 0.0.0 02/06/2015 [15646.323311] RIP: 0010:shrink_slab_memcg+0x9f/0x1e0 [15646.323320] Code: 48 63 45 ac 45 31 f6 49 8b 84 c4 b8 0f 00 00 4c 8b b8 e0 03 00 00 4d 85 ff 0f 84 2f 01 00 00 48 63 35 29 ad 43 02 49 8b 7f 18 bc cf 37 00 39 05 1a ad 43 02 89 c3 7f 71 e9 10 01 00 00 8b 40 [15646.323321] RSP: 0000:ffffa3b5c0c9fc08 EFLAGS: 00000286 [15646.323324] RAX: ffff8f19a0782800 RBX: ffffa3b5c0c9fe00 RCX: 8000000000000007 [15646.323325] RDX: ffff8f1a4113ddc0 RSI: 00000000000004ec RDI: ffff8f1a61346820 [15646.323326] RBP: ffffa3b5c0c9fc70 R08: 0000000000000005 R09: ffffa3b5c0c9fe00 [15646.323327] R10: ffff8f19a0782800 R11: ffff8f19a6bac000 R12: ffff8f19a6bac000 [15646.323328] R13: ffff8f1a7ffd6000 R14: 0000000000000000 R15: ffff8f1a61344000 [15646.323329] FS: 0000000000000000(0000) GS:ffff8f1a74600000(0000) knlGS:0000000000000000 [15646.323330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15646.323331] CR2: 00007f962c772798 CR3: 000000011b474003 CR4: 00000000003706f0 [15646.323335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15646.323336] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15646.323337] Call Trace: [15646.323338] [15646.323341] shrink_slab+0x10a/0x120 [15646.323344] ? shrink_slab+0x10a/0x120 [15646.323346] shrink_node_memcgs+0x188/0x1d0 [15646.323349] shrink_node+0x16e/0x600 [15646.323351] balance_pgdat+0x36e/0x810 [15646.323354] ? try_to_del_timer_sync+0x53/0x90 [15646.323358] kswapd+0x10c/0x1c0 [15646.323360] ? balance_pgdat+0x810/0x810 [15646.323362] kthread+0x12a/0x150 [15646.323366] ? set_kthread_struct+0x50/0x50 [15646.323368] ret_from_fork+0x22/0x30 [15646.323375] [15646.323376] NMI backtrace for cpu 2 [15646.323379] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.15.0-72-generic #79-Ubuntu [15646.323382] Hardware name: Nutanix AHV, BIOS 0.0.0 02/06/2015 [15646.323384] RIP: 0010:psi_group_change+0x19f/0x420 [15646.323389] Code: 00 83 f9 02 0f 85 b1 00 00 00 45 8b 6c 24 08 45 85 ed 0f 95 c2 84 d2 74 06 89 f2 d3 e2 09 d3 83 f8 07 74 40 83 c0 01 8d 48 ff <83> f9 03 0f 84 b8 00 00 00 76 c3 83 f9 05 0f 84 f1 00 00 00 83 f9 [15646.323391] RSP: 0018:ffffa3b5c0890d68 EFLAGS: 00000002 [15646.323394] RAX: 0000000000000002 RBX: 0000000000000001 RCX: 0000000000000001 [15646.323395] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000000 [15646.323397] RBP: ffffa3b5c0890dc0 R08: 00000e3af18ab2cc R09: 0000000000000001 [15646.323398] R10: 0000000000000002 R11: 0000000000000001 R12: ffffc3b5b8698a00 [15646.323400] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000001 [15646.323401] FS: 0000000000000000(0000) GS:ffff8f1a74680000(0000) knlGS:0000000000000000 [15646.323403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15646.323405] CR2: 000000000184fd00 CR3: 000000012c2da001 CR4: 00000000003706e0 [15646.323409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15646.323414] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15646.323416] Call Trace: [15646.323417] [15646.323419] psi_task_change+0x9e/0x140 [15646.323422] enqueue_task+0x15a/0x1a0 [15646.323427] ttwu_do_activate+0x58/0xf0 [15646.323431] try_to_wake_up+0x1ed/0x5a0 [15646.323434] ? enqueue_task_fair+0x18a/0x6a0 [15646.323440] ? __hrtimer_init+0x110/0x110 [15646.323444] wake_up_process+0x15/0x20 [15646.323447] hrtimer_wakeup+0x22/0x40 [15646.323450] __hrtimer_run_queues+0x107/0x230 [15646.323453] ? kvm_clock_get_cycles+0x11/0x20 [15646.323458] hrtimer_interrupt+0x101/0x220 [15646.323461] __sysvec_apic_timer_interrupt+0x61/0xe0 [15646.323465] sysvec_apic_timer_interrupt+0x7b/0x90 [15646.323470] [15646.323471] [15646.323472] asm_sysvec_apic_timer_interrupt+0x1b/0x20 [15646.323475] RIP: 0010:native_safe_halt+0xb/0x10 [15646.323478] Code: 2c ff 5b 41 5c 41 5d 5d c3 cc cc cc cc 4c 89 ee 48 c7 c7 80 45 a5 97 e8 b3 61 8d ff eb ca cc eb 07 0f 00 2d e9 7d 45 00 fb f4 cc cc cc cc eb 07 0f 00 2d d9 7d 45 00 f4 c3 cc cc cc cc cc 0f [15646.323480] RSP: 0018:ffffa3b5c00b7e78 EFLAGS: 00000206 [15646.323482] RAX: ffffffff969b5f70 RBX: ffff8f1a40363e80 RCX: 0000000000000000 [15646.323484] RDX: 0000000002ce3d1d RSI: ffffa3b5c00b7e40 RDI: 0000000002ce3d1e [15646.323485] RBP: ffffa3b5c00b7e80 R08: 0000000000000001 R09: 0000000000000007 [15646.323486] R10: 0000000000000000 R11: 0000000000000006 R12: 0000000000000000 [15646.323488] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15646.323489] ? __cpuidle_text_start+0x8/0x8 [15646.323493] ? default_idle+0xe/0x20 [15646.323495] arch_cpu_idle+0x15/0x20 [15646.323500] default_idle_call+0x3e/0xd0 [15646.323502] cpuidle_idle_call+0x179/0x1e0 [15646.323505] do_idle+0x83/0xf0 [15646.323507] cpu_startup_entry+0x20/0x30 [15646.323509] start_secondary+0x12a/0x180 [15646.323512] secondary_startup_64_no_verify+0xc2/0xcb [15646.323518] [15646.323520] NMI backtrace for cpu 3 [15646.323522] CPU: 3 PID: 3382609 Comm: podman Not tainted 5.15.0-72-generic #79-Ubuntu [15646.323524] Hardware name: Nutanix AHV, BIOS 0.0.0 02/06/2015 [15646.323525] RIP: 0033:0x46ee3a [15646.323528] Code: 6c 24 30 48 8d 6c 24 30 0f 1f 84 00 00 00 00 00 48 85 c0 0f 8e 0f 01 00 00 48 89 44 24 40 4c 89 74 24 28 49 8b 8e 70 01 00 00 <48> 85 c9 75 39 48 8d 05 1a 20 3a 01 e8 75 9a fa ff 48 8b 4c 24 28 [15646.323529] RSP: 002b:000000c00036a268 EFLAGS: 00000202 [15646.323531] RAX: 0000000002faf080 RBX: 000000000274db18 RCX: 000000c000090140 [15646.323532] RDX: 0000000001721a00 RSI: 0000000000000000 RDI: 000000c0000001a0 [15646.323533] RBP: 000000c00036a298 R08: 0000000000000001 R09: 0000000000000000 [15646.323534] R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000006 [15646.323535] R13: 000000000199d740 R14: 000000c0000001a0 R15: 0000000000000000 [15646.323536] FS: 00007f9165ffb640 GS: 0000000000000000 ```
Luap99 commented 1 year ago

The first step is to always test is with the latest podman and kernel. Reading the links you gave this is a kernel bug which was fixed a while ago so you to update your kernel.

RHEL always has "older" kernels they just backport many bug fixes so it is fixed there. So you have to update your kernel or ask ubuntu to backport the required fixes