search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.68k stars 2.41k forks source link

"Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd" #6084

Closed avikivity closed 4 years ago

avikivity commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. podman run -it --rm fedora:32

Describe the results you received:

Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

Describe the results you expected:

#

Additional information you deem important (e.g. issue happens only occasionally):

Happens all the time

Output of podman version:

Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.14.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.14.2
  podmanVersion: 1.9.1
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.15-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 33da5ef83bf2abc7965fc37980a49d02fdb71826'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: tmp.scylladb.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.7-300.fc32.x86_64
  memFree: 5275238400
  memTotal: 33541488640
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.fc32.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  swapFree: 16869486592
  swapTotal: 16869486592
  uptime: 93h 3m 6.55s (Approximately 3.88 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/avi/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/avi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/avi/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.9.1-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Fully updated Fedora 32.

avikivity commented 4 years ago

Update: this started to happen on my second Fedora 32 machine. On the other hand, I rebooted the first one (due to an unrelated problem :( ) and the problem went away.

I'm running Linux 5.6.7 on the still-broken machine, so it's unlikely to be kernel version related.

mheon commented 4 years ago

Can you provide ~/.config/containers/libpod.conf from a system that reproduces?

avikivity commented 4 years ago

That one has no ~/.config/containers/libpod.conf.

$ cat ~/.config/containers/libpod.conf
cat: /home/avi/.config/containers/libpod.conf: No such file or directory
$ podman run -it --rm fedora:32
Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd
avikivity commented 4 years ago

Linux avi 5.6.7-300.fc32.x86_64 #1 SMP Thu Apr 23 14:13:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

mheon commented 4 years ago

It's definitely a v2 system from podman info, so the issue must be the cgroup manager. Can you try a container with --cgroup-manager=systemd and see if that works?

mheon commented 4 years ago

@rhatdan Does this look like a potential containers.conf issue to you? Maybe a default resource limit specified in the config?

hiisukun commented 4 years ago

I also bumped into this. First time running podman on Fedora 32, trying to launch a couple of containers.

Output:

[hiisukun@serv:~]$ podman run -it --rm fedora:32
Trying to pull registry.fedoraproject.org/fedora:32...
Getting image source signatures
Copying blob 3088721d7dbf done  
Copying config d81c91deec done  
Writing manifest to image destination
Storing signatures
Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

podman version:

Version:            1.9.1
RemoteAPI Version:  1
Go Version:         go1.14.2
OS/Arch:            linux/amd64

podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.14.2
  podmanVersion: 1.9.1
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.15-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 33da5ef83bf2abc7965fc37980a49d02fdb71826'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: serv
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.8-300.fc32.x86_64
  memFree: 3510607872
  memTotal: 14608658432
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.fc32.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  swapFree: 7417098240
  swapTotal: 7423913984
  uptime: 50h 46m 55.42s (Approximately 2.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/hiisukun/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/hiisukun/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /home/hiisukun/.local/share/containers/storage/volumes

rpm -q podman:

podman-1.9.1-1.fc32.x86_64

uname -a:

Linux omni 5.6.8-300.fc32.x86_64 #1 SMP Wed Apr 29 19:01:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

libpod.conf:

[hiisukun@serv:~]$ cat ~/.config/containers/libpod.conf
cat: /home/hiisukun/.config/containers/libpod.conf: No such file or directory

Other:

hiisukun commented 4 years ago

Well, after shutting down the various things and rebooting, it has indeed gone away as per @avikivity mentioning. Hopefully the source of the problem can be fixed since rebooting isn't always convenient.

avikivity commented 4 years ago

It's definitely a v2 system from podman info, so the issue must be the cgroup manager. Can you try a container with --cgroup-manager=systemd and see if that works?

Tried it, no change.

avikivity commented 4 years ago

Since I needed the machine to work I rebooted it (unfortunately, couldn't check if logout is sufficient since Fedora 32 loses the keyboard/mouse after logout). The problem went away on that machine too. Unfortunately this means I can't help with debugging it until it recurs again (and then I will be limited by needing podman to work).

MartinNowak commented 4 years ago

For me it was an outdated ~/.config/containers/libpod.conf config file that still had a cgroup_manager = "cgroupfs" line. I don't remember having created that config file, so I suspect it was automatically created by podman (or maybe by me following some tutorial). I removed that file and now running a container works without the --cgroup-manager=systemd override. This doesn't sound exactly like OPs problem, but seems related enough to post here.

mheon commented 4 years ago

The reboot part of this seems bizarre...

If anyone else encounters this problem: can you try running podman system reset and see if things go back to normal after that?

trebor8x commented 4 years ago

I am facing the same issue after upgrading Fedora 31 to 32. Only config I could find is located at /usr/share/containers/libpod.conf and it states cgroup_manager = "systemd"

Running with debug output yields:

podman run --log-level=debug --rm -it debian bash
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs
DEBU[0000] Ignoring lipod.conf EventsLogger setting "journald". Use containers.conf if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] container-default [] private [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL  ...
...
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
...

Running podman system reset does not change the result.

Previously I removed the directory .local/share/containers/, but the error persists.

mheon commented 4 years ago

Can you provide full output for the command with debugging enabled?

Can you try manually setting --log-driver=k8s-file and seeing if that resolves the issue?

Finally, does systemctl --user as the user running Podman work?

trebor8x commented 4 years ago

The command systemctl --user reports: Failed to connect to bus. It is also impossible to start a new gnome-terminal by now.

mheon commented 4 years ago

Alright, I don't think this is Podman. It sounds like the systemd user session is just not running, but starts after a reboot? Tempted to say it's a systemd bug?

denesb commented 4 years ago

I'm also seeing this on Fedora 31.

avikivity commented 4 years ago

@mheon please change the error message to "could not connect to dbus" then. The current error message looks like it's trying to help, but it's actively misleading.

mheon commented 4 years ago

Digging deeper, that part looks like a bug. We should be discarding resource limits and not erroring unless manually set, but it looks like containers.conf is forcing resources to be set on every Podman invocation.

@rhatdan From a brief parse, I'm pretty sure containers.conf is unconditionally setting the PID limit, which is causing us to blow up on cgroups v2 systems that are not using the systemd cgroup manager.

amarshall commented 4 years ago

I see the original error as well when using cgroup_manager="cgroupfs" (which I need to use due to a different error which I’ll likely open an issue for soon). Downgrading podman from 1.9.1 to 1.8.2 appears to resolve it.

numas commented 4 years ago

I experienced this too on my Fedora 31 laptop (been through 28 > 29 > 30 > 31 upgrades). An inelegant deletion of ~/.config/containers fixed it, but not sure if there are consequences in this.

rhatdan commented 4 years ago

Most likely you are fine. Podman will just use the defaults if that directory was removed. If you made any customizations, then there could be issues.

Navdeepuniyal commented 4 years ago

I am facing the same issue when using REST API:

curl --header "Content-Type: application/json"   --request POST   --data '{
>   "image": "nginx",
>   "privileged": true,
>   "publish_image_ports": true,
>   "name": "rem-nginx"
> }' http://localhost:4041/containers/create

Error:

{"cause":"invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd","message":"CreateContainerFromCreateConfig(): invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd","response":500}

Any hints?

rhatdan commented 4 years ago

Did you remove the libpod.conf, and could you try out podman-1.9.2 in updates-testing?

Navdeepuniyal commented 4 years ago

Interestingly, it works with the restart.

andrewgdunn commented 4 years ago

Seeing the same issue on two fresh F32 instances, rolling back podman-1.8.2 (whats currently "new" in F31) was our current solution. We'll be watching this ticket with intent!

rhatdan commented 4 years ago

I would love to see if people with this issue, see if it is fixed by podman 1.9.2

mheon commented 4 years ago

I think this one is resolved by 1.9.2, but I'm going to hold off on closing until someone can confirm

andrewgdunn commented 4 years ago

I think this one is resolved by 1.9.2, but I'm going to hold off on closing until someone can confirm

Do you know when 1.9.2 will be promoted out of testing on F32?

rhatdan commented 4 years ago

Did you give it good karma?

limburgher commented 4 years ago

I was hitting this and 1.9.2 fixes it for me, but I can't find the update in bodhi to provide karma.

rhatdan commented 4 years ago

https://bodhi.fedoraproject.org/updates/FEDORA-2020-fb351fcdbb

limburgher commented 4 years ago

Thanks. :)

isimluk commented 4 years ago

Works for me. Karma voted. Thanks. :sake:

mheon commented 4 years ago

Thanks for checking, everyone! Glad to see this is resolved.

andrewgdunn commented 4 years ago

I was finally able to consume 1.9.2 on F32. After a podman system reset I'm still seeing this error. I'm instancing the container with a unit file:

[Unit]
Description=Podman running deluge
Wants=network.target
After=network-online.target

[Service]
WorkingDirectory=/app/acquisition/deluge
User=acquisition
Group=acquisition
Restart=no
ExecStartPre=/usr/bin/rm -f %T/%N.pid %T/%N.cid
ExecStartPre=/usr/bin/podman rm --ignore -f deluge
ExecStart=/usr/bin/podman run --conmon-pidfile %T/%N.pid --cidfile %T/%N.cid --cgroups=no-conmon \
  --name=deluge \
  --publish 127.0.0.1:8112:8112 \
  --security-opt label=disable \
  --volume /app/acquisition/deluge/config:/config \
  docker.io/linuxserver/deluge:latest
ExecStop=/usr/bin/podman stop --ignore deluge -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f deluge
ExecStopPost=/usr/bin/rm -f %T/%N.pid %T/%N.cid
KillMode=control-group
Type=simple

[Install]
WantedBy=multi-user.target default.target

A downgrade to 1.8.2 and I can run again without issues. This is a fresh install of F32 as of a couple days ago. No changing of any configuration related to containers.

mheon commented 4 years ago

Do you have a ~/config/containers/libpod.conf and if so, can you provide the contents of it?

RheaAyase commented 4 years ago

The error is still present.

RheaAyase commented 4 years ago

@mheon should this issue be reopened or opened as a new issue? (The history here is pretty good, I'd rather keep it.) Even if it's a systemd bug, to serve as a tracker and to point people in the right direction.

mheon commented 4 years ago

I think we should probably re-open until a patch can be merged to improve the error message.

On Sun, May 24, 2020, 13:03 Radka Gustavsson notifications@github.com wrote:

@mheon https://github.com/mheon should this issue be reopened or opened as a new issue? (The history here is pretty good, I'd rather keep it.) Even if it's a systemd bug, to serve as a tracker and to point people in the right direction.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/6084#issuecomment-633260732, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3AOCHNFBB5ZE2TU6AEQTLRTFHONANCNFSM4MZJ27NQ .

andrewgdunn commented 4 years ago

The error is still present.

* `podman-1.9.2-1.fc32.x86_64`

* `~/config/containers/libpod.conf` does not exist

* `/usr/share/containers/libpod.conf` says it's systemd (default and unchanged)

* `$ systemctl --user` -> `Failed to connect to bus: No such file or directory`

* `$ podman system reset` does not solve anything either.

* Not restarting the whole space station (semi-production server) to get this to work.

Correct, my system errors with the same state that @RheaAyase enumerates above.

rhatdan commented 4 years ago

/usr/share/containers/libpod.conf should also not be there. We don't want to be shipping libpod.conf at all anymore. rpm -qf /usr/share/containers/libpod.conf podman-1.9.2-1.fc32.x86_64

That should be removed.

sharmay commented 4 years ago

Removed rm /usr/share/containers/libpod.conf Verified ls -l /usr/share/containers/libpod.conf

ls: cannot access '/usr/share/containers/libpod.conf': No such file or directory

reboot system (test system)

podman run -it --rm fedora:32

Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

rpm -q podman

podman-1.9.2-1.fc32.x86_64
mheon commented 4 years ago

@rhatdan From my reading of the issue, we've probably fixed the libpod.conf issues in containers/common, but now we're giving this error in cases where the user does not have a systemd user session enabled as well, and it's not very helpful there.

@sharmay Can you check if systemctl --user works?

sharmay commented 4 years ago

I was switching user sudo su - <user> and for this new user systemctl --user is not working. This is where I was running podman.

For now enabled ssh and podman working.

jasimmk commented 4 years ago

I started experimenting with podman.. Getting some inconsistencies on running as non root user

On first run, getting a permission denied error

[user@localhost ~]$ podman run -it hello-world
Error: sd-bus call: Permission denied: OCI runtime permission denied error

Running podman as root works fine

[user@localhost ~]$ sudo su
[sudo] password for user:
[root@localhost user]# podman run -it hello-world

Hello from Docker!
.....

After exiting root, on second time as user

[user@localhost ~]$ podman run -it hello-world
Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd

System information

[user@localhost ~]$ uname -a
Linux localhost 5.6.13-300.fc32.x86_64 #1 SMP Thu May 14 22:51:37 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Podman Info

[user@localhost ~]$ podman info --debug
debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.14.2
  podmanVersion: 1.9.2
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.16-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.16, commit: 1044176f7dd177c100779d1c63931d6022e419bd'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.13-300.fc32.x86_64
  memFree: 24427372544
  memTotal: 25190215680
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.fc32.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  swapFree: 0
  swapTotal: 0
  uptime: 3m 17.12s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /tmp/run-1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes

Libpod config locations 

[user@localhost ~]$ locate libpod.conf
/usr/share/containers/libpod.conf
/usr/share/man/man5/libpod.conf.5.gz

Podman installation

[user@localhost ~]$ sudo yum distro-sync --enablerepo=updates-testing podman
[user@localhost ~]$ sudo yum -y install podman
[user@localhost ~]$ rpm -q podman
podman-1.9.2-1.fc32.x86_64
mheon commented 4 years ago

Do you have a systemd user session available? IE, does systemctl --user work?

jasimmk commented 4 years ago

Do you have a systemd user session available? IE, does systemctl --user work?

$ systemctl --user
Failed to connect to bus: No such file or directory
mheon commented 4 years ago

That would be your problem then. Your distro may not enabled the systemd user session by default, or you may be logging in by a method that does not create one (su or sudo). You can also make a ~/.config/containers/libpod.conf with the line cgroup_manager = "cgroupfs" in it - that will disable the ability to set resource limits on rootless containers, but should remove the issue.

jasimmk commented 4 years ago

Do you have a systemd user session available? IE, does systemctl --user work?

$ systemctl --user Failed to connect to bus: No such file or directory

Lol, that was a good hint. I searched for this 

Failed to connect to bus: No such file or directory

and landed on https://bbs.archlinux.org/viewtopic.php?id=234813

It says. I need to ensure UsePAM yes in /etc/ssh/sshd_config, Which I had actually set to UsePAM no as part of securing sshd. In Fedora 32 /etc/ssh/sshd_config there is a clear warning though.

# WARNING: 'UsePAM no' is not supported in Fedora and may cause several
# problems.

I enabled back 'UsePAM yes' and restarted system. 

$ podman run -it hello-world

Hello from Docker!

Everything works fine !!!! 🏆