Unable to build a container image with Podman inside a running Docker container

[thomassuedbroecker]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Unable to build a container image with Podman inside a running Docker container

Steps to reproduce the issue:

1. Build a container image with podman installed and other tools installed using Docker CE Here you find the Dockerfile I used on github.

     docker build -t "tsuedbroecker/cns-workshop-tools:v5" .`

2. Run the created Docker image with the installed podman as a container in Docker

    docker run -it --name demo "tsuedbroecker/cns-workshop-tools:v5" /bin/bash

3. Clone a github project into the running Container and navigate to a folder with the Dockerfile we want to use

     root@39aa78c1cb7a:/tmp #
     git clone https://github.com/IBM/cloud-native-starter
     cd cloud-native-starter/author-java-jee

4. Execute podman build inside the running container

root@39aa78c1cb7a:/tmp/cloud-native-starter/authors-java-jee#  podman build -t authors:v1 .

Describe the results you received:

The build stops where the RUN commands inside the Dockerfile are starting.

ERRO[0000] unable to write system event: "write unixgram @00022->/run/systemd/journal/socket: sendmsg: no such file or directory" 
STEP 1: FROM maven:3.5-jdk-8
Getting image source signatures
Copying blob a587a86c9dcb done  
Copying blob e5c3f8c317dc done  
Copying blob 193a6306c92a done  
Copying blob a4c7ee7ef122 done  
Copying blob a7c0dad691e9 done  
Copying blob bc9ab73e5b14 done  
Copying blob 367a6a68b113 done  
Copying blob 60c0e52d1ec2 done  
Copying blob c9d22bc43935 done  
Copying blob 41dbcd970503 done  
Copying blob 72c52bb7e9b7 done  
Copying blob 275d6d4a835d done  
Copying config 985f3637de done  
Writing manifest to image destination
Storing signatures
STEP 2: COPY src /usr/src/app/src
--> cf11e165dcd
STEP 3: COPY pom.xml /usr/src/app
--> f766f151d8a
STEP 4: RUN mvn -f /usr/src/app/pom.xml clean package
error running container: error creating new mount namespace for [/bin/sh -c mvn -f /usr/src/app/pom.xml clean package]: operation not permitted
STEP 5: FROM open-liberty:19.0.0.9-kernel-java11
ERRO[0110] unable to write build event: "write unixgram @00022->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Error: error building at STEP "RUN mvn -f /usr/src/app/pom.xml clean package": error while running runtime: exit status 1

Describe the results you expected:

The image should be builded.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: 39aa78c1cb7a
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.39-linuxkit
  linkmode: dynamic
  memFree: 312197120
  memTotal: 2083184640
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 1015566336
  swapTotal: 1073737728
  uptime: 3h 16m 3.38s (Approximately 0.12 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus: {}
  imageStore:
    number: 4
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 2.1.1~2 amd64 [installed]
podman/unknown 2.1.1~2 arm64
podman/unknown 2.1.1~2 armhf
podman/unknown 2.1.1~2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): Running Docker container with docker desktop community Version: 2.5.0.0 (49427) Channel: stable

[mheon]

@rhatdan I'm pretty sure we have a blog somewhere about builds in locked-down containers, but I can't seem to locate it. Would you happen to know where it lives?

[thomassuedbroecker]

@mheon @rhatdan I tried different resources, but I end up in all paths with the same problem you find here.

[mheon]

I think the recommendation for builds specifically is to use buildah with the chroot isolation. @nalind Do we have any documentation on that?

[nalind]

Not as such AFAIK, but the default images we provide set BUILDAH_ISOLATION to "chroot" in the environment and configure storage.conf to use the "overlay" driver with "mount_program=/usr/bin/fuse-overlayfs", which requires that the container be started with --device /dev/fuse or an equivalent. Prior to fuse-overlayfs being an option, we were using the "vfs" driver, which doesn't require any special privileges.

[thomassuedbroecker]

@nalind hmm, that sounds a bit like the new error I get now.

I tried out using Fedora with my new Dockerfile Fedora.Dockerfile. I was hoping maybe that will solve the problem and I did same steps.

command:

podman build -t authors:v1 .

But that is the result: output:

Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver

[thomassuedbroecker]

@nalind @nalind @rhatdan Maybe you know it, where can I find a simple Dockerfile template with the right settings to build a container image, in which I can execute one of these commands?

podman build -t myimage:v1 .

or

buildah bud -t myimage:v1 .

Do you have a link to such Dockerfile? That would be awesome ...

[rhatdan]

I wrote an article on this last year:

https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/