search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.32k stars 2.38k forks source link

Podman 2.2.0 is broken on Ubuntu Focal #8539

Closed ocafebabe closed 3 years ago

ocafebabe commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. Install Podman 2.2.0 using documented method/repo: https://podman.io/getting-started/installation

  2. Execute: podman info

Describe the results you received:

podman -v
podman version 2.2.0
podman info
error creating temporary file: No such file or directory
Error: error setting up the process: open /tmp/podman-run-1001/libpod/pause.pid: no such file or directory

Describe the results you expected:

Info output...

Additional information you deem important (e.g. issue happens only occasionally):

It works with sudoer accounts...

Output of podman version:

podman version
error creating temporary file: No such file or directory
Error: error setting up the process: open /tmp/podman-run-1001/libpod/pause.pid: no such file or directory

Output of podman info --debug:

podman info --debug
error creating temporary file: No such file or directory
Error: error setting up the process: open /tmp/podman-run-1001/libpod/pause.pid: no such file or directory

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.2.0~1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

physical

mheon commented 3 years ago

Can you provide the output of podman info --log-level=debug?

ocafebabe commented 3 years ago

There you go: podman-info-debug.txt

jdockter commented 3 years ago

Adding mine here as well podman-info-debug-jpd.txt

mheon commented 3 years ago

@ocafebabe It seems like yours indicates Podman is working?

The one from @jdockter indicates that the directory we're using for the pause process is not the actual temporary directory - Podman is configured to use /tmp/run-1000/libpod/tmp as a temporary files directory, but it's looking for the pause process at /tmp/podman-run-1000/libpod/pause.pid (note the run-1000 in the first, versus podman-run-1000 in the second). @giuseppe PTAL

ocafebabe commented 3 years ago

@mheon well, like I said in my bug report, it works with "sudo" but not as a regular user...

mheon commented 3 years ago

@ocafebabe The directories in that debug log you provided certainly look rootless? And aside from the added debug logs, it seems to have run exactly as expected.

mheon commented 3 years ago

Can you try other commands with `--log-level=debug and see if they also work?

ocafebabe commented 3 years ago

I tried the same command (podman info --log-level=debug) but this time as a non-sudoer user and the output looks very similar as the one provided by @jdockter: podman-info-debug-2.txt

I get the same error regarding the tmp directory: 

error creating temporary file: No such file or directory
Error: error setting up the process: open /tmp/podman-run-1001/libpod/pause.pid: no such file or directory
lsm5 commented 3 years ago

@ocafebabe can you try 2.2.0~2 from https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman

ocafebabe commented 3 years ago

@ocafebabe can you try 2.2.0~2 from https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman

I just did but I get the same error...

allurefx commented 3 years ago

@ocafebabe It seems like yours indicates Podman is working?

The one from @jdockter indicates that the directory we're using for the pause process is not the actual temporary directory - Podman is configured to use /tmp/run-1000/libpod/tmp as a temporary files directory, but it's looking for the pause process at /tmp/podman-run-1000/libpod/pause.pid (note the run-1000 in the first, versus podman-run-1000 in the second). @giuseppe PTAL

Thanks! Creating a symlink from /tmp/run-1000 to /tmp/podman-run-1000 fixed the problem for me, but clearly this is a kludge...is there something we need to do our side for a permanent fix or will this be fixed in the next podman update?

mheon commented 3 years ago

This is definitely a bug on our end and we'll see about getting it fixed in a 2.2.1 soon.

On Tue, Dec 1, 2020, 19:14 Sekhar Ravinutala notifications@github.com wrote:

@ocafebabe https://github.com/ocafebabe It seems like yours indicates Podman is working?

The one from @jdockter https://github.com/jdockter indicates that the directory we're using for the pause process is not the actual temporary directory - Podman is configured to use /tmp/run-1000/libpod/tmp as a temporary files directory, but it's looking for the pause process at /tmp/podman-run-1000/libpod/pause.pid (note the run-1000 in the first, versus podman-run-1000 in the second). @giuseppe https://github.com/giuseppe PTAL

Thanks! Creating a symlink from /tmp/run-1000 to /tmp/podman-run-1000 fixed the problem for me, but clearly this is a kludge...is there something we need to do our side for a permanent fix or will this be fixed in the next podman update?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/podman/issues/8539#issuecomment-736902270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3AOCDAN7EJA6SN6N32IXTSSWBHFANCNFSM4UJLQXLQ .

azmandzamiri commented 3 years ago

I am seeing the same issue but with a different file name . Is there a fix available ?

Setting up podman (2.2.0~2) ...

Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

Processing triggers for libc-bin (2.27-3ubuntu1.2) ...

error creating temporary file: No such file or directory

Error: error setting up the process: open /tmp/podman-run-2000/libpod/pause.pid: no such file or directory

error creating temporary file: No such file or directory

Error: error setting up the process: open /tmp/podman-run-2000/libpod/pause.pid: no such file or directory
jdockter commented 3 years ago

Now I'm seeing this error in our builds with Ubuntu 18. It was passing just 3 hours earlier today.

Warning: apt-key output should not be parsed (stdout is not a terminal)

2020-12-02 03:28:45 URL:https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04/Release.key [1093/1093] -> "-" [1]

OK

Selecting previously unselected package catatonit.

(Reading database ... 97126 files and directories currently installed.)

Preparing to unpack .../00-catatonit_0.1.5~1_amd64.deb ...

Unpacking catatonit (0.1.5~1) ...

Selecting previously unselected package conmon.

Preparing to unpack .../01-conmon_2.0.20~1_amd64.deb ...

Unpacking conmon (2.0.20~1) ...

Selecting previously unselected package containers-golang.

Preparing to unpack .../02-containers-golang_0.6.0~1_all.deb ...

Unpacking containers-golang (0.6.0~1) ...

Selecting previously unselected package containers-image.

Preparing to unpack .../03-containers-image_5.8.1~1_all.deb ...

Unpacking containers-image (5.8.1~1) ...

Selecting previously unselected package containers-common.

Preparing to unpack .../04-containers-common_1.2.0~6_all.deb ...

Unpacking containers-common (1.2.0~6) ...

Selecting previously unselected package libyajl2:amd64.

Preparing to unpack .../05-libyajl2_2.1.0-2build1_amd64.deb ...

Unpacking libyajl2:amd64 (2.1.0-2build1) ...

Selecting previously unselected package crun.

Preparing to unpack .../06-crun_0.14.1~3_amd64.deb ...

Unpacking crun (0.14.1~3) ...

Selecting previously unselected package libgpgme11:amd64.

Preparing to unpack .../07-libgpgme11_1.10.0-1ubuntu2_amd64.deb ...

Unpacking libgpgme11:amd64 (1.10.0-1ubuntu2) ...

Selecting previously unselected package containernetworking-plugins.

Preparing to unpack .../08-containernetworking-plugins_0.8.7~1_amd64.deb ...

Unpacking containernetworking-plugins (0.8.7~1) ...

Selecting previously unselected package runc.

Preparing to unpack .../09-runc_1.0.0~rc10-0ubuntu1~18.04.2_amd64.deb ...

Unpacking runc (1.0.0~rc10-0ubuntu1~18.04.2) ...

Selecting previously unselected package podman-plugins.

Preparing to unpack .../10-podman-plugins_1.1.1~1_amd64.deb ...

Unpacking podman-plugins (1.1.1~1) ...

Selecting previously unselected package podman.

Preparing to unpack .../11-podman_2.2.0~2_amd64.deb ...

Unpacking podman (2.2.0~2) ...

Selecting previously unselected package slirp4netns.

Preparing to unpack .../12-slirp4netns_1.1.7~1_amd64.deb ...

Unpacking slirp4netns (1.1.7~1) ...

Selecting previously unselected package varlink.

Preparing to unpack .../13-varlink_19~1_amd64.deb ...

Unpacking varlink (19~1) ...

Setting up catatonit (0.1.5~1) ...

Setting up runc (1.0.0~rc10-0ubuntu1~18.04.2) ...

Setting up slirp4netns (1.1.7~1) ...

Setting up containers-image (5.8.1~1) ...

Setting up libgpgme11:amd64 (1.10.0-1ubuntu2) ...

Setting up libyajl2:amd64 (2.1.0-2build1) ...

Setting up containers-golang (0.6.0~1) ...

Setting up crun (0.14.1~3) ...

Setting up conmon (2.0.20~1) ...

Setting up containers-common (1.2.0~6) ...

Adding 'diversion of /etc/containers/storage.conf.orig to /etc/containers/storage.conf.orig.containers-orig by containers-common'

Adding 'diversion of /etc/containers/policy.json.orig to /etc/containers/policy.json.orig.containers-orig by containers-common'

Adding 'diversion of /etc/containers/registries.d/default.yaml.orig to /etc/containers/registries.d/default.yaml.orig.containers-orig by containers-common'

Adding 'diversion of /etc/containers/containers.conf.orig to /etc/containers/containers.conf.orig.containers-orig by containers-common'

Setting up containernetworking-plugins (0.8.7~1) ...

Setting up varlink (19~1) ...

Setting up podman-plugins (1.1.1~1) ...

Setting up podman (2.2.0~2) ...

Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

Processing triggers for libc-bin (2.27-3ubuntu1.2) ...

[INFO] podman version: 

error creating temporary file: No such file or directory

Error: error setting up the process: open /tmp/podman-run-2000/libpod/pause.pid: no such file or directory
afbjorklund commented 3 years ago

Seems to be some kind of residue after: 3daef2e82638a7bdae5037d4fc0d8918a57368e3 (v2.1.0-529-g3daef2e82)

--- a/pkg/util/utils_supported.go
+++ b/pkg/util/utils_supported.go
@@ -38,7 +38,7 @@ func GetRuntimeDir() (string, error) {
                        }
                }
                if runtimeDir == "" {
-                       tmpDir := filepath.Join(os.TempDir(), fmt.Sprintf("run-%s", uid))
+                       tmpDir := filepath.Join(os.TempDir(), fmt.Sprintf("podman-run-%s", uid))
                        if err := os.MkdirAll(tmpDir, 0700); err != nil {
                                logrus.Debug(err)
                        }

DEBU[0000] Using tmp dir /tmp/run-1000/libpod/tmp

Error: error setting up the process: open /tmp/podman-run-1000/libpod/pause.pid: no such file or directory

But uses /run/user/1000 under systemd (XDG_RUNTIME_DIR).

ocafebabe commented 3 years ago

Thanks to @afbjorklund comment, I was able to found another workaround:

sudo loginctl enable-linger mypodmanuser
sudo -u mypodmanuser -i
export XDG_RUNTIME_DIR=/run/user/$UID

In my case, XDG_RUNTIME_DIR isn't set by pam systemd because I'm using "sudo" and apparently this isn't supported yet.

afbjorklund commented 3 years ago

That explains why it was working on my Ubuntu 20.04 desktop (with systemd)

DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp

Seems that it sneaked back into using vfs, wonder where fuse-overlayfs went ?

Shouldn't that be in the podman dependencies ? At least under recommends ?

cgroupManager: cgroupfs

eventLogger: journald

  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev

graphDriverName: vfs

Added #8560 for overlay, and cgroups v1 was the reason for cgroupfs (in rootless)

So far it seems happy with docker runc (1.0.0-rc10) instead of crio runc (v1.0.0-rc92)

jdockter commented 3 years ago

So what is the best workaround at the moment? Or is 2.2.1 when we see this fixed?

mheon commented 3 years ago

For the moment, symlink /tmp/podman-run-$UID to /tmp/run-$UID (replacing $UID with the UID of the user running Podman).

I think we're probably just missing an MkdirAll somewhere to ensure that directory exists on reboot.

adelton commented 3 years ago

For our GitHub Action and Travis CI, the symlink approach was good enough as workaround: https://github.com/freeipa/freeipa-container/commit/de1738252dafc01b4400276eb85b6af70134059c

mheon commented 3 years ago

Alright, I think I've figured this one out, and it's unfortunately somewhat more complicated than it seems. @rhatdan altered the way we get our default runtime path for systems not using systemd as PID1 (and without XDG_RUNTIME_DIR) in https://github.com/containers/podman/commit/3daef2e82638a7bdae5037d4fc0d8918a57368e3 (thanks for the pointer @afbjorklund). This only takes effect for new installs, though - existing ones will continue to use older paths cached in the database to ensure that existing containers are not broken. Problem: the code for determining the rootless pause process path does not take the database into account, so it's not using the cached paths and is trying to use the new path on systems where it's not being created because they are still using /tmp/run-1000. I think the best course of action is probably to take the pause process path computation, and require it to use the Libpod runtime directory, but I need to determine if we have that path available every time we compute it.

mheon commented 3 years ago

I made #8556 to resolve this. Would greatly appreciate testing from anyone willing to build from source.

adelton commented 3 years ago

For the record, I see the issue on CI's where podman is always installed afresh, so I'm not sure about the old cached data.

jdockter commented 3 years ago

Symlink workaround, ln -s /tmp/run-$( id -u ) /tmp/podman-run-$( id -u ) worked in our Travis build, thanks!

blavoie commented 3 years ago

Don't know if it's related, but applying symlink workaround on a fresh install leads to another error:

$ podman pull alpine
ERRO[0000] cannot mkdir /var/lib/aidboxdb/rundir/libpod: mkdir /var/lib/aidboxdb/rundir/libpod: no such file or directory

Manually creating this directory solves the issue.

EDIT: /var/lib/aidboxdb/ is $HOME

Forceu commented 3 years ago

I am having the same problem. For me a temporary workaround is

ln -s /tmp/run-$( id -u ) /tmp/podman-run-$( id -u )
mkdir /tmp/podman-run-$( id -u )/libpod
mpolanski commented 3 years ago

@mheon I've tested https://github.com/containers/podman/pull/8556/commits/e8cecd7ca803d68837b5c74eca2703cfafb2221b on Alpine Linux and it works fine. I'm waiting with the upgrade until it's merged to master or 2.2.1 is out: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/15142.

c-goes commented 3 years ago

The workaround with mkdir is working in WSL2. ln is not enough.

mheon commented 3 years ago

For reference, we are planning a 2.2.1 on Monday that will include the fix - wanted a few extra days to get more fixes for 2.2 issues landed.

asottile commented 3 years ago

for me it's a different directory, but similar error message:

$ sudo -u otheruser podman run --rm ubuntu:focal echo hi
ERRO[0000] cannot mkdir /home/otheruser/rundir/libpod: mkdir /home/otheruser/rundir/libpod: no such file or directory
rahulagrawalpsl commented 3 years ago

Hey Guys, I am still getting this error for podman 2.2.0. Should i create a different issue for this ?

error setting up the process: open /tmp/podman-run-2000/libpod/pause.pid: no such file or directory

mheon commented 3 years ago

That's the same issue. Please try the symlink workaround mentioned earlier (link /tmp/podman-run-2000 to /tmp/run-2000) until Podman 2.2.1 can be released (probably later today)

jdockter commented 3 years ago

@mheon when the 2.2.1 release hits kubic, will we need to remove the symlink workaround? Just trying to reduce any downtime we might have in our Travis builds.

mheon commented 3 years ago

It should not be necessary - we'll go back to using the old path. The symlink will no longer be used, but it existing won't harm anything.

Also, it should be rolling out soon (tag is pushed, I believe OBS is building)