search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.16k stars 2.36k forks source link

podman inside podman #9566

Closed DavidMachacek closed 3 years ago

DavidMachacek commented 3 years ago

/kind bug

Description I would like to start Ubuntu based image for building and running containers inside pipeline.

Steps to reproduce the issue:

  1. Build base container using podman build -f inception-Dockerfile -t host-container . Dockerfile

    FROM ubuntu
# tz data non interactive
RUN apt-get update && ln -fs /usr/share/zoneinfo/Europe/Prague /etc/localtime
RUN apt-get install -y \
btrfs-progs \
git \
golang-go \
go-md2man \
iptables \
libassuan-dev \
libbtrfs-dev \
libc6-dev \
libdevmapper-dev \
libglib2.0-dev \
libgpgme-dev \
libgpg-error-dev \
libprotobuf-dev \
libprotobuf-c-dev \
libseccomp-dev \
libselinux1-dev \
libsystemd-dev \
pkg-config \
runc \
uidmap
RUN apt-get install -y curl && \
apt-get -y install sudo
RUN apt-get install -y libvshadow-utils    
# add repo with latest kubic podman release
RUN useradd -m --uid 1000 podman && \
echo "podman     ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER 1000 
# sourcing ". /etc/os-release" does not work, setting up versionId manually
ENV VERSION_ID=20.04
RUN  echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /"  | \
sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
RUN curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key  | \
sudo apt-key add -
RUN sudo apt-get update && \
sudo apt-get -y upgrade && \
sudo apt-get -y install podman
RUN sudo apt-get install -y podman
# set vfs
ENV STORAGE_DRIVER=vfs
ENTRYPOINT ["bash"]`

  2. Run the container using podman run -v mycontainers:/var/lib/containers -ti host-container For some reason I have to pass volume with mounted overlayFS for host-container (by using something like mount -t overlay -o lowerdir=base,upperdir=diff,workdir=workdir overlayfs overlay) otherwise I get ERROR ERRO[0000] 'overlay' is not supported over <unknown> at "/var/lib/containers/storage/overlay" Error: kernel does not support overlay fs: 'overlay' is not supported over <unknown> at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

  3. Alter /etc/containers/storage.conf to enable VFS (setting driver="vfs" and deleting mountopt)

  4. Inside host-container start any other container, i.e. sudo podman run -ti fedora --privileged --host=net

Describe the results you received:

ERRO[0003] unable to write pod event: "write unixgram @002b6->/run/systemd/journal/socket: sendmsg: no such file or directory"
ERRO[0003] Error preparing container 1228648b7c8de2407fee7e222c98ccb0cacde38dc6acdf1acd44b1e0429f7d5b: error creating network namespace for container 1228648b7c8de2407fee7e222c98ccb0cacde38dc6acdf1acd44b1e0429f7d5b: mount --make-rshared /run/netns failed: "operation not permitted"
Error: failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/1228648b7c8de2407fee7e222c98ccb0cacde38dc6acdf1acd44b1e0429f7d5b/userdata/shm": operation not permitted

Describe the results you expected: Running container inside host-container

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 1
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: edf59cfcf3c4
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-1038-aws
  linkmode: dynamic
  memFree: 76091392
  memTotal: 1026154496
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 2h 46m 37.41s (Approximately 0.08 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 0
    stopped: 15
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus: {}
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.1

Package info (apt list podman):

Listing... Done
podman/unknown,now 100:3.0.1-2 amd64 [installed]
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? Yes

Additional environment details (AWS, VirtualBox, physical, etc.): AWS backed Ubuntu 20:04

Other problem I have to use sudo for every podman command. Otherwise I get 

podman@edf59cfcf3c4:/$ podman version
Error: cannot setup namespace using newuidmap: exit status 1

/proc/self/uid_map: 0 1000 1 1 100000 65536

/proc/self/gid_map 0 1000 1 1 100000 65536

Questions 1) Why cant I run new container inside my podman host-container? 2) How can I avoid having to prepare overlayFS volume for host-container? 3) Why do I get Error: cannot setup namespace using newuidmap error without sudo?

mheon commented 3 years ago

@rhatdan Got a Podman-in-Podman bug for you and @umohnani8

rhatdan commented 3 years ago

@DavidMachacek Have you looked at https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/

DavidMachacek commented 3 years ago

@rhatdan there is link to stable Dockerfile which no longer exists, so I went for this one. I built it buildah bud -f Dockerfile -t builder and ran podman run -v ./build:/build:z -v /var/lib/containers1:/var/lib/containers:Z --device /dev/fuse -ti builder, but when I was inside it, I installed podman dnf install -y podman and then start podman version, but got following error Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted. I was succesfull in building image inside it using buildah (from Dockerfile inside ./build folder), but I am still looking for a solution to run container inside container without Docker-in-Docker, not to just build them, in order to i.e. run component test with multiple containers (ie. start service container against another DB container) in my pipelines (without them I dont want to push untested container image to registry). I watched several of your talks on youtube too but didnt find the answer. I acknowledge this topic might not be a bug, but still I would love to see my podman container inception running :) BTW: I am a big fan of your work! 👍

rhatdan commented 3 years ago

You need to use the fuse-overlay driver for now.

DavidMachacek commented 3 years ago

@rhatdan I am using it. podman run -v test:/var/lib/containers:Z --device /dev/fuse -ti --log-level debug new-container ends with DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: imagestore=/var/lib/shared DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted

rhatdan commented 3 years ago

@giuseppe PTAL

giuseppe commented 3 years ago

If /usr/bin/newuidmap and /usr/bin/newgidmap are installed as setuid binaries you need to add CAP_SYS_ADMIN.

If they use file capabilities as on Fedora, CAP_SETUID and CAP_SETGID are enough.

DavidMachacek commented 3 years ago

@giuseppe Ubuntu 20:04 step 1) podman run -v /var/lib/containers1:/var/lib/containers:Z --device /dev/fuse -ti --cap-add CAP_SYS_ADMIN my-container based on builder Inside Fedora-based container as rootless user: step 1) sudo dnf install -y podman step 2) podman run --net=host --events-backend=file fedora returns Error: cannot setup namespace using newuidmap: exit status 1 I reinstalled shadow-utils and chmod 4755 /usr/bin/newu(g)idmap with no help, getcap /usr/bin/newuidmap /usr/bin/newgidmap returns /usr/bin/newuidmap cap_setuid=ep /usr/bin/newgidmap cap_setgid=ep

When I do it as root, I get Error: cannot chown /var/lib/containers/storage/overlay/c9448f1effa1206f7251da933500d757753df04b7bc6b611797190156a17ec4e/merged to 0:0: chown /var/lib/containers/storage/overlay/c9448f1effa1206f7251da933500d757753df04b7bc6b611797190156a17ec4e/merged: operation not permitted

github-actions[bot] commented 3 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 3 years ago

We have just published

https://www.redhat.com/sysadmin/podman-inside-container And https://www.redhat.com/sysadmin/podman-inside-kubernetes

Please read these and see if they help solve your problem. Reopen if you need more information.