search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.44k stars 785 forks source link

Cannot build images with podman build 1.8.0 in Docker containers #2175

Closed mathstuf closed 4 years ago

mathstuf commented 4 years ago

Description

podman build fails with:

error running container: error creating container for [/bin/sh -c sh /root/install_deps.sh]: cannot open sd-bus: No such file or directory

The container is not systemd-as-pid1, so enabling the journal doesn't work. Passing --events-backend=file also doesn't avoid the issue.

Full output (including dep resolution for podman itself): https://gitlab.kitware.com/ben.boeckel/cmb-superbuild/-/jobs/4816230

Steps to reproduce the issue:

  1. Use gitlab-ci to run a container (fedora:31 in my case)
  2. Try and use podman build

Describe the results you received:

Failure to connect to sd-bus.

Describe the results you expected:

A built image

Output of podman version if reporting a podman build issue:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

*Output of `cat /etc/release`:**

Fedora release 31 (Thirty One)
NAME=Fedora
VERSION="31 (Thirty One)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Thirty One)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Fedora release 31 (Thirty One)
Fedora release 31 (Thirty One)

Output of uname -a:

Taken from the host of the builder in question.

Linux dovim 5.3.11-100.fc29.x86_64 #1 SMP Tue Nov 12 20:41:25 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

Fedora default configuration.

TomSweeneyRedHat commented 4 years ago

@mathstuf I think this is the same issue a: https://bugzilla.redhat.com/show_bug.cgi?id=1768954

From that, if you can, please try:

@giuseppe any other thoughts?

paleozogt commented 4 years ago

I've tried the workarounds. While they've gotten me further along, its still not working:

➜  docker run --rm -it --privileged quay.io/podman/stable:latest
[root@eb3fa2d5bf40 /]# podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64
[root@eb3fa2d5bf40 /]# printf "FROM ubuntu\nRUN cat /etc/*release\n" > Dockerfile
[root@eb3fa2d5bf40 /]# sed -e 's/sss//g' -i /etc/nsswitch.conf
[root@eb3fa2d5bf40 /]# podman --cgroup-manager=cgroupfs build -t foo .
STEP 1: FROM ubuntu
Getting image source signatures
Copying blob f9a83bce3af0 done
Copying blob de83a2304fa1 done
Copying blob b6b53be908de done
Copying blob 423ae2b273f4 done
Copying config 72300a873c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN cat /etc/*release
2020-02-28T23:16:35.000368398Z: executable file not found in $PATH: No such file or directory
error running container: error creating container for [/bin/sh -c cat /etc/*release]: : exit status 1
Error: error building at STEP "RUN cat /etc/*release": error while running runtime: exit status 1
paleozogt commented 4 years ago

There's no 1.8.0 image available (due to this), but if I manually update podman inside the container to 1.8.0, it works!

➜  docker run --rm -it --privileged quay.io/podman/stable:latest
[root@bb2093099153 /]# podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64
[root@bb2093099153 /]# yum update -y podman

...

[root@bb2093099153 /]# podman version
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

[root@bb2093099153 /]# printf "FROM ubuntu\nRUN cat /etc/*release\n" > Dockerfile
[root@bb2093099153 /]# sed -e 's/sss//g' -i /etc/nsswitch.conf
[root@bb2093099153 /]# podman --cgroup-manager=cgroupfs build -t foo .
ERRO[0000] unable to write system event: "write unixgram @00bf1->/run/systemd/journal/socket: sendmsg: no such file or directory"
STEP 1: FROM ubuntu
Getting image source signatures
Copying blob b6b53be908de done
Copying blob f9a83bce3af0 done
Copying blob de83a2304fa1 done
Copying blob 423ae2b273f4 done
Copying config 72300a873c [======================================] 3.3KiB / 3.3KiB
Writing manifest to image destination
Storing signatures
STEP 2: RUN cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
STEP 3: COMMIT foo
237bb203dc815e0a9bc56209e6e822792829b15f148fb2c2eb6126d10f021cc6
237bb203dc815e0a9bc56209e6e822792829b15f148fb2c2eb6126d10f021cc6

[root@bb2093099153 /]# podman images
REPOSITORY                 TAG      IMAGE ID       CREATED          SIZE
localhost/foo              latest   237bb203dc81   13 seconds ago   66.6 MB
docker.io/library/ubuntu   latest   72300a873c2c   7 days ago       66.6 MB
h1dden-da3m0n commented 4 years ago

Okay it seems i have the same issue, yet it still persists even with v1.8.0. After forcing the update to 1.8.0 via yum i get the following error using the example from @paleozogt 

[root@66d0aa0fb67e ~]#  podman --cgroup-manager=cgroupfs build -t foo .
STEP 1: FROM ubuntu
STEP 2: RUN cat /etc/*release
Error: error building at STEP "RUN cat /etc/*release": error mounting container "1bb40fa7899f4d698d5fe4c9592144e55697aa49f2c87269717ac82fbd2ee4c5": error mounting build container "1bb40fa7899f4d698d5fe4c9592144e55697aa49f2c87269717ac82fbd2ee4c5": failed to canonicalise path for /var/lib/containers/storage/overlay/a50e8d976ab30165352c31692764efb560adf4b81fcdd2b7d9ed701f401ee78b/merged: lstat /var/lib/containers/storage/overlay/a50e8d976ab30165352c31692764efb560adf4b81fcdd2b7d9ed701f401ee78b/merged: invalid argument

I used the exact same steps as @paleozogt on a local test REHL 7.x server (with docker 1.13.1) and on PWD (play-with-docker, i had no other test machine and windows was note really an option), still resulted in the same error. I sadly can't pinpoint the source of this error nor do I fully understand the error message I'm afraid :-S

(sorry if this a different issue and I should have opened a new issue, I shall open on if deemed so)

EDIT: I might want to add that i get the same error with buildah (if i try to build the same Dockerfile via buildah bud .)

mathstuf commented 4 years ago

--cgroup-manager=cgroupfs worked for me (no need to mess with sssd settings). Thanks.

paleozogt commented 4 years ago

I'm seeing both @h1dden-da3m0n's result as well as @mathstuf's, depending on how I run it.

On a regular bare-metal host (Ubuntu Bionic), if I run docker directly it'll work as in my comment above.

However, if I run from within a container (such as a kubernetes pod or play-with-docker), it'll fail. Here's it failing on play-with-docker:

[root@91f7962db48e /]# docker run --rm -it --privileged quay.io/podman/stable:latest
[root@91f7962db48e /]# podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

[root@91f7962db48e /]# yum update -y podman
...
[root@91f7962db48e /]# podman version
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

[root@91f7962db48e /]# printf "FROM ubuntu\nRUN cat /etc/*release\n" > Dockerfile
[root@91f7962db48e /]# sed -e 's/sss//g' -i /etc/nsswitch.conf
[root@91f7962db48e /]# podman --cgroup-manager=cgroupfs build -t foo .
ERRO[0000] unable to write system event: "write unixgram @001ac->/run/systemd/journal/socket: sendmsg: no such file or directory" 
STEP 1: FROM ubuntu
Getting image source signatures
Copying blob de83a2304fa1 done  
Copying blob b6b53be908de done  
Copying blob 423ae2b273f4 done  
Copying blob f9a83bce3af0 done  
Copying config 72300a873c done  
Writing manifest to image destination
Storing signatures
STEP 2: RUN cat /etc/*release
Error: error building at STEP "RUN cat /etc/*release": error mounting container "ddc666e827fa83f74d5b817ce4b3dfbae05e33d68577428fe6ae053f198fa872": error mounting build container "ddc666e827fa83f74d5b817ce4b3dfbae05e33d68577428fe6ae053f198fa872": failed to canonicalise path for /var/lib/containers/storage/overlay/08588a7c7ee18f8f1b24a311690e17df11ff5e0681fa58af9b6dcab2828191fe/merged: lstat /var/lib/containers/storage/overlay/08588a7c7ee18f8f1b24a311690e17df11ff5e0681fa58af9b6dcab2828191fe/merged: invalid argument
mathstuf commented 4 years ago

Hmm. I'm seeing it work within Docker images on a Fedora 31 host (executed via Docker due to needing the Docker socket for gitlab-runner).

paleozogt commented 4 years ago

I've been digging into this some more.

On k8s clusters with Ubuntu Bionic hosts, I can run a pod that is able to use podman.

However, on k8s clusters with Centos 7 hosts, running a pod using podman will see the failed to canonicalise path error. This seems similar to @h1dden-da3m0n's comment about seeing the problem on REHL 7.x.

What features/OS version does buildah/podman need on the host system in order to work?

rhatdan commented 4 years ago

RHEL 7.* < 7.8 can not run rootless containers with fuse-overlay. So this can cause some issues. As of 7.8, RHEL7 should work better with rootless environments.

h1dden-da3m0n commented 4 years ago

I'll hav a look which REHL 7.x version we have deployed on moday and report back.

paleozogt commented 4 years ago

I've got Centos 7.4.1708 and 7.6.1810, neither of which work. So this tracks with @rhatdan's comment.

h1dden-da3m0n commented 4 years ago

okay, just checked we are running REHL 7.7, so is it really just a Host OS version issue?

rhatdan commented 4 years ago

Can we close this issue, and wait for RHEL7.8 or even better, move to rhel8? That is where the cool kids are going... :^)

mathstuf commented 4 years ago

I'm fine with closing it; a solution seems to have been found for everyone's issues here.

And FYI, RHEL 8 needs a podman release with containers/libpod#4499 in a stable enough state before I can deploy with it for container usage (though the official Docker repos are also available I suppose, I don't like adding repos like that unless necessary).

h1dden-da3m0n commented 4 years ago

I thought REHL 8.1 was the one for cool kids 😁 , but fine bye me to close this. Even though it is kinda frustrating -.- (just one last side-node: Are you/we sure this cannot be reproduced in REHL7.8 / 8+ ? I can't test that, as we do not have access to even a test REHL 8+ machine ☹️ )

rhatdan commented 4 years ago

I am thoroughly confused by this issue. I was responding to a few comments back that was attempting to execute rootless podman on a RHEL/Centsos 7.4 box.

h1dden-da3m0n commented 4 years ago

Not sure if I was the confusing factor for @rhatdan or not (sorry if I was indeed), but as far as I see it Buildah/Podman has some problem running inside of a container.

In this issue there are the following host OSes listed to face the same issue as the OP as well as an issue that seems to pop up as soon as you pass the OP issue:

The OP error seems to get "fixed" once you update podman / buildah inside of the container to the latest version (1.8.0 for podman I think), yet that then leads to the failed to canonicalise path for ... error which I barged in with (yet I faced the OP error first, which lead me posting it in here)

I think I "summarized" the issue so far, hope this helps clear up the confusion. Finally, thank you all for the interesting insight and I hope this simply gets resolved by a host OS version upgrade (yet god knows when that is going to drop in my case 🙄)

edit: Okay, I now properly read through the issue again and finally spotted my error of understanding this issue by re-reading the comment by @rhatdan

RHEL 7.* < 7.8 can not run rootless containers with fuse-overlay. So this can cause some issues. As of 7.8, RHEL7 should work better with rootless environments.

The key words here being containers with fuse-overlay and that such containers wont work on REHL < 7.8 (or 8). I also seem to have misread a comment from @paleozogt, who clearly stated that his Debian host worked fine, as well as a comment from @mathstuf, who stated that a Fedora 31 host worked for fine.

Finally, I want to apologise for the holdup on closing the issue and maybe confusing the one or the other participant.

TLDR: RHEL 7.* < 7.8 can not run rootless containers with fuse-overlay

yajo commented 4 years ago

This is happening to me while running inside Gitlab's free runner. Is there any solution? :confused:

TomSweeneyRedHat commented 4 years ago

@Yajo can you confirm the versions of Podman that are in use and were you able to use any of the workarounds listed in the comments above? If not, can you talk a little bit about your environment, especially OS versions and the such.

yajo commented 4 years ago

can you confirm the versions of Podman that are in use

Yes, you can read that in the attached log, but let me summarize:

 (1/29): conmon-2.0.15-1.fc31.x86_64.rpm         109 kB/s |  40 kB     00:00    
(2/29): containers-common-0.2.0-1.fc31.x86_64.r 695 kB/s |  49 kB     00:00    
(3/29): catatonit-0.1.5-2.fc31.x86_64.rpm       527 kB/s | 300 kB     00:00    
(4/29): crun-0.13-2.fc31.x86_64.rpm             1.1 MB/s | 162 kB     00:00    
(5/29): fuse-overlayfs-1.0.0-1.fc31.x86_64.rpm  920 kB/s |  65 kB     00:00    
(6/29): libslirp-4.1.0-2.fc31.x86_64.rpm        957 kB/s |  67 kB     00:00    
(7/29): iptables-1.8.3-7.fc31.x86_64.rpm        3.2 MB/s | 466 kB     00:00    
(8/29): podman-plugins-1.9.1-1.fc31.x86_64.rpm  9.2 MB/s | 1.4 MB     00:00    
(9/29): runc-1.0.0-102.dev.gitdc9208a.fc31.x86_  16 MB/s | 2.7 MB     00:00    
(10/29): containernetworking-plugins-0.8.5-1.fc  10 MB/s |  11 MB     00:01    
(11/29): slirp4netns-1.0.1-1.fc31.x86_64.rpm    697 kB/s |  49 kB     00:00    
(12/29): fuse-common-3.6.2-1.fc31.x86_64.rpm     56 kB/s | 7.8 kB     00:00    
(13/29): fuse3-3.6.2-1.fc31.x86_64.rpm          702 kB/s |  54 kB     00:00    
(14/29): podman-1.9.1-1.fc31.x86_64.rpm          19 MB/s |  13 MB     00:00    
(15/29): criu-3.13-5.fc31.x86_64.rpm            1.6 MB/s | 492 kB     00:00    
(16/29): fuse3-libs-3.6.2-1.fc31.x86_64.rpm     986 kB/s |  92 kB     00:00    
(17/29): kmod-26-4.fc31.x86_64.rpm              3.4 MB/s | 122 kB     00:00    
(18/29): libbsd-0.9.1-4.fc31.x86_64.rpm         1.8 MB/s | 101 kB     00:00    
(19/29): libmnl-1.0.4-10.fc31.x86_64.rpm        947 kB/s |  28 kB     00:00    
(20/29): libnetfilter_conntrack-1.0.7-3.fc31.x8 1.9 MB/s |  63 kB     00:00    
(21/29): libnet-1.1.6-18.fc31.x86_64.rpm        1.1 MB/s |  64 kB     00:00    
(22/29): libnfnetlink-1.0.1-16.fc31.x86_64.rpm  987 kB/s |  30 kB     00:00    
(23/29): jansson-2.12-4.fc31.x86_64.rpm         249 kB/s |  43 kB     00:00    
(24/29): libnftnl-1.1.3-2.fc31.x86_64.rpm       2.0 MB/s |  81 kB     00:00    
(25/29): libnl3-3.5.0-1.fc31.x86_64.rpm         7.1 MB/s | 331 kB     00:00    
(26/29): libvarlink-util-18-2.fc31.x86_64.rpm   817 kB/s |  52 kB     00:00    
(27/29): nftables-0.9.1-3.fc31.x86_64.rpm       4.1 MB/s | 315 kB     00:00    
(28/29): protobuf-c-1.3.1-3.fc31.x86_64.rpm     514 kB/s |  35 kB     00:00    
(29/29): yajl-2.1.0-13.fc31.x86_64.rpm          720 kB/s |  39 kB     00:00
yajo commented 4 years ago

were you able to use any of the workarounds listed in the comments above?

Oh sorry forgot this.

I cannot use the container as privileged, because it's running in public free CI runners, which let you specify the image where to run from and it is executed through docker. You can see the full .gitlab-ci.yml definition, so it's 100% reproducible. There's no option to pass --privileged flag to the container that runs the CI job.

So, podman is trying to run inside a fedora image (I think it still resolves to fedora:31 by default) executed by Docker.

So my case is podman-in-docker, and it's not something I can escape from :confused:

If I can help any further, just tell me :blush:

mathstuf commented 4 years ago

You're just going to need privileged rights inside of a Docker runner to use podman. If the CI infra doesn't offer it, you'll need a different solution I'm afraid. You could stand your own runners up, but then you're probably waiting on other gitlab bugs/feature requests related to that.

I think you're missing something anyways, but maybe this is just the way it manifests:

Error: cannot open sd-bus: No such file or directory: OCI runtime command not found error

mathstuf commented 4 years ago

Ah, that is the error message at the beginning here. Maybe the cgfoupfs option works?

yajo commented 4 years ago

OK I have done a very simple test: Inside Gitlab's free runbot offering (based on Docker, unprivileged), just run podman --cgroup-manager cgroupfs container run --rm -it hello-world. The result is:

 Error: error creating libpod runtime: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver

See the full log: https://gitlab.com/yajoman/rpi_cooler/-/jobs/569793360

Is there anything else I could do to make it work? Maybe using VFS storage driver? :thinking:

giuseppe commented 4 years ago

Is there anything else I could do to make it work? Maybe using VFS storage driver? thinking

I'd suggest you try with fuse-overlayfs. fuse-overlayfs on top of overlay should work fine

paleozogt commented 4 years ago

I'd suggest you try with fuse-overlayfs. fuse-overlayfs on top of overlay should work fine

What does that podman command look like?

yajo commented 4 years ago

Did I do anything wrong? See https://gitlab.com/yajoman/rpi_cooler/-/jobs/575997744:

 $ podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world
time="2020-06-01T17:34:52Z" level=error msg="Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage"
Error: error creating libpod runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supported
TomSweeneyRedHat commented 4 years ago

@giuseppe PTAL

rhatdan commented 4 years ago

$ podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world

fuse-overlayfs is not a graph driver, this should be overlay. And you should enable the fuse-overlay line in the /etc/containers/storage.conf inside of the container image.

You should also specify podman command with

podman --device /dev/fuse --cgroup-manager ...

That will get you closer.

yajo commented 4 years ago

Yikes, no way.

I discovered that gitlab shared runners are actually privileged, so you can reproduce locally easily with this:

➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world
ERRO[0000] Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage 
Error: error creating libpod runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supported

It seems --device is a wrong flag:

➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --device /dev/fuse --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world
Error: unknown flag: --device

Also enabling the fuse-overlay line doesn't seem to help:

➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora bash -c "sed -i 's@#mount_program = \"/usr/bin/fuse-overlayfs\"@mount_program = \"/usr/bin/fuse-overlayfs\"@' /etc/containers/storage.conf && podman --cgroup-manager cgroupfs --storage-driver fuse-overlayfs container run --rm -it hello-world"
ERRO[0000] Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage 
Error: error creating libpod runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supported

There must be a way... :thinking: Any ideas?

yajo commented 4 years ago

I see VFS works out of the box, but it's too much not-optimized, mostly compared to overlay:

➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-driver vfs container run --rm -it hello-world
Trying to pull registry.fedoraproject.org/hello-world...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/hello-world...
  name unknown: Repo not found
Trying to pull registry.centos.org/hello-world...
  manifest unknown: manifest unknown
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done  
Copying config bf756fb1ae done  
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

No way to do it with fuse-overlayfs?

rhatdan commented 4 years ago

--storage-driver=fuse-overlayfs is wrong, it should be --storage-driver=overlay. You need to uncomment fuse-overlayfs line in /etc/containers/storage.conf in the container image and that will make the overlay driver use the helper program fuse-overlay. fuse-overlayfs needs to have a /dev/fuse within the container to work.

yajo commented 4 years ago

Still not working:

➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlayfs container run --device /dev/fuse --rm -it hello-world
ERRO[0000] Failed to GetDriver graph overlayfs /var/lib/containers/storage 
Error: error creating libpod runtime: failed to GetDriver graph overlayfs /var/lib/containers/storage: driver not supported

➤ docker container run --device /dev/fuse --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlayfs container run --device /dev/fuse --rm -it hello-world
ERRO[0000] Failed to GetDriver graph overlayfs /var/lib/containers/storage 
Error: error creating libpod runtime: failed to GetDriver graph overlayfs /var/lib/containers/storage: driver not supported

Are you sure this is supposed to work? :thinking:

mathstuf commented 4 years ago

It's working for me. Host is CentOS 7 or 8 if that helps.

rhatdan commented 4 years ago

overlayfs->overlay overlayfs driver does not exists.

yajo commented 4 years ago

:tada: :rocket: Made it work, thanks!

➤ docker container run --rm --privileged -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlay container run --rm -it hello-world
Trying to pull registry.fedoraproject.org/hello-world...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/hello-world...
  name unknown: Repo not found
Trying to pull registry.centos.org/hello-world...
  manifest unknown: manifest unknown
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done  
Copying config bf756fb1ae done  
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

I didn't even need --device /dev/fuse.

I also modified --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs

It works on Gitlab CI too: https://gitlab.com/yajoman/rpi_cooler/-/jobs/582567308 :heart:

yajo commented 4 years ago

Nice, now another question. If I remove --privileged from docker, it fails:

➤ docker container run --rm -it registry.gitlab.com/yajoman/ci-tools/fedora podman --cgroup-manager cgroupfs --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs --storage-driver overlay container run --device /dev/fuse --rm -it hello-world
Error: error creating libpod runtime: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted

Is there a way to replace --privileged with some specific --cap-add so at least there's less privileges? Or that doesn't make sense for this use case?

rhatdan commented 4 years ago

--cap-add SYS_ADMIN Will probably work. But I would just get rid of Docker altogether. :^)

rhatdan commented 4 years ago

You might have to disable SELinux if you are using it and seccomp. --security-opt seccomp=unconfined --security-opt label=disabled

rhatdan commented 4 years ago

We have added some docs on this, closing this issue.

paleozogt commented 4 years ago

@rhatdan can you link to the docs?

rhatdan commented 4 years ago

https://developers.redhat.com/blog/2019/04/04/build-and-run-buildah-inside-a-podman-container/ https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/