search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.98k stars 2.34k forks source link

cannot clone : Operation not permitted and Error: cannot re-exec process #10797

Closed sachinkaushik closed 3 years ago

sachinkaushik commented 3 years ago

Hi Team,

I'm trying to create a rootless container using below Dockerfile. I have created a user called CLI and that is a rootless user for container. When I ran container and trying hit podman commands like podman images or podman ps. I'm getting error mentioned in subject line.

Can you please help me what else configuration required in Dockerfile to create a rootless container.

----------------------------------------------------Dockerfile Start------------------------------------

FROM quay.io/podman/stable:latest

RUN useradd cli RUN touch /etc/subgid /etc/subuid \ && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ && echo cli:100000:65536 > /etc/subuid \ && echo cli:100000:65536 > /etc/subgid

RUN yum install -y \ python3-pip \ python3 python3-wheel

RUN pip install jupyterlab

USER cli WORKDIR /home/cli

ENTRYPOINT ["jupyter", "lab", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

----------------------------------------------------Dockerfile End------------------------------------------------------------------------ Is there any Dockerfile example, that can help me to create a rootless container.

podman-error

Thank you in advance.

rhatdan commented 3 years ago

So you are attempting to run a rootless container within a rootless container?

If so then the range of UIDs within the rootless container must be a subset of the UIDs the user is using.

Most likely your rootless user is only allowed to use UIDs 0-65000, but you are attempting to run with UID 100000, which is not mapped within your user namespace. The example we use: https://github.com/containers/podman/blob/master/contrib/podmanimage/stable/Dockerfile

Allocates 5000 UIDs starting at UID 2000.

RUN useradd podman; \
echo podman:10000:5000 > /etc/subuid; \
echo podman:10000:5000 > /etc/subgid;
sachinkaushik commented 3 years ago

Thank you for your response..!!

Yes I'm trying to create a rootless container and inside that container, User can build container image using podman. I used the Dockerfile, from that link which you have shared and integrated few things related to jupyter lab. But still I'm getting an error saying "Error: error creating runtime static files directory: mkdir /home/podman/.local/share/containers/storage: permission denied" .

Can you please check and let me know what else config still required to make it work.

-----------------------------Dockerfile Start--------------------------------

FROM registry.fedoraproject.org/fedora:latest

# Don't include container-selinux and remove
# directories used by yum that are just taking
# up space.
RUN dnf -y update; yum -y reinstall shadow-utils; \
yum -y install podman fuse-overlayfs --exclude container-selinux; \
rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman; \
echo podman:10000:5000 > /etc/subuid; \
echo podman:10000:5000 > /etc/subgid;

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers
RUN mkdir -p /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf
ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

RUN chown podman:podman -R /home/podman

# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN yum install -y \
            python3-pip \
            python3 python3-wheel 

RUN pip install jupyterlab

COPY login-script.sh /etc/containers/
RUN chmod -R 777 /etc/containers/login-script.sh

USER podman
WORKDIR /home/podman

ENTRYPOINT ["/etc/containers/login-script.sh"]

------------------------------------------Dockerfile End---------------------------------------------------

Error :

podman-error-command

podman-ps

rhatdan commented 3 years ago

What podman command are you running to create the container?

lmgray commented 3 years ago

seems like this is a more general podman-3.2.x problem -- working on #10802 I found I can't run basic podman commands even in images from quay.io (no docker build involved) since 3.1.2 was replaced by 3.2.x:

$ docker run quay.io/podman/stable:v3.1.2 podman --storage-driver=vfs version
Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.15.8
Built:        Thu Apr 22 13:21:33 2021
OS/Arch:      linux/amd64

$ docker run quay.io/podman/stable:v3.2.0 podman --storage-driver=vfs version
cannot clone: Operation not permitted
Error: cannot re-exec process

$ docker run quay.io/podman/stable:v3.2.1 podman --storage-driver=vfs version
cannot clone: Operation not permitted
Error: cannot re-exec process
sachinkaushik commented 3 years ago

Hi @rhatdan ,

I'm using below docker run command to create the container.

docker run --name pod -d -p 8888:8888 pod

pod

--------------------------------------Dockerfile Start----------------------------------------------- FROM registry.fedoraproject.org/fedora:latest

RUN dnf -y update; yum -y reinstall shadow-utils; \ yum -y install podman fuse-overlayfs --exclude container-selinux; \ rm -rf /var/cache /var/log/dnf /var/log/yum.

RUN useradd podman; \ echo podman:10000:5000 > /etc/subuid; \ echo podman:10000:5000 > /etc/subgid;

VOLUME /var/lib/containers

RUN mkdir -p /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

chmod containers.conf and adjust storage.conf to enable Fuse storage.

RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage./a "/var/lib/shared",' -e 's|^mountopt[[:space:]]=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN yum install -y \ python3-pip \ python3 python3-wheel

RUN pip install jupyterlab

COPY login-script.sh /etc/containers/ RUN chmod -R 777 /etc/containers/login-script.sh RUN chown podman:podman -R /home/podman/ USER podman

VOLUME /home/podman/.local/share/containers WORKDIR /home/podman

ENTRYPOINT ["/etc/containers/login-script.sh"] -----------------------------------------Dockerfile End------------------------------------------------------------

sachinkaushik commented 3 years ago

Hi @rhatdan ,

Some how Im able to run podman command inside container but now Im getting below error.

using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first fuse-overlayfs: cannot mount: No such file or directory

mount error

podman info :: host: arch: amd64 buildahVersion: 1.21.0 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 16 distribution: distribution: fedora version: "34" eventLogger: file hostname: cliservice-6f9c59bdc4-r9qks idMappings: gidmap:

  • container_id: 0 host_id: 1000 size: 1
  • container_id: 1 host_id: 10000 size: 5000 uidmap:
  • container_id: 0 host_id: 1000 size: 1
  • container_id: 1 host_id: 10000 size: 5000 kernel: 4.18.0-240.22.1.el8_3.x86_64 linkmode: dynamic memFree: 58096541696 memTotal: 67193081856 ociRuntime: name: crun package: crun-0.20.1-1.fc34.x86_64 path: /usr/bin/crun version: |- crun version 0.20.1 commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /tmp/podman-run-1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.9-1.fc34.x86_64 version: |- slirp4netns version 1.1.8+dev commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.0 swapFree: 0 swapTotal: 0 uptime: 39m 29.88s registries: search:
  • registry.fedoraproject.org
  • registry.access.redhat.com
  • docker.io
  • quay.io store: configFile: /home/podman/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.5.0-1.fc34.x86_64 Version: |- fusermount3 version: 3.10.4 fuse-overlayfs: version 1.5 FUSE library version 3.10.4 using FUSE kernel interface version 7.31 graphRoot: /home/podman/.local/share/containers/storage graphStatus: Backing Filesystem: overlayfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 1 runRoot: /tmp/podman-run-1000/containers volumePath: /home/podman/.local/share/containers/storage/volumes version: APIVersion: 3.2.1 Built: 1623697949 BuiltTime: Mon Jun 14 19:12:29 2021 GitCommit: "" GoVersion: go1.16.3 OsArch: linux/amd64 Version: 3.2.1
rhatdan commented 3 years ago

Add the --device /dev/fuse to Docker or Podman command line

sachinkaushik commented 3 years ago

Hi @rhatdan ,

Thanks for quick response..!!

I tried below command, but still same error

podman build --device /dev/fuse -t demo . Error: error creating build executor: error getting info of source device /dev/fuse: stat /dev/fuse: no such file or directory

rhatdan commented 3 years ago

Ok, you are confusing me. You keep switching back and forth on questions.

Let's concentrate on one thing at a time.

Are we set with running podman run within a docker run container?

rhatdan commented 3 years ago

The comment on the --device /dev/fuse is for the docker/podman command on the outside launching the podman within the container.

docker run --device /dev/fuse ... podman build ... podman run --device /dev/fuse ... podman build ...

Although we would recommend that you just use buildah bud --isolation=chroot for this use case.

https://developers.redhat.com/blog/2019/04/04/build-and-run-buildah-inside-a-podman-container

sachinkaushik commented 3 years ago

@rhatdan ,

I have created a rootless container using below Dockerfile. And now inside that container, I'm trying to create a container image using podman build -t demo . command. When I'm running this command that time only getting below error related to mount.

Command : podman build -t demo .

Error :

mount error

--------------------------------------------------Dockerfile start----------------------------------------------- FROM registry.fedoraproject.org/fedora:latest

RUN dnf -y update; yum -y reinstall shadow-utils; \ yum -y install podman fuse-overlayfs --exclude container-selinux; \ rm -rf /var/cache /var/log/dnf /var/log/yum.

RUN yum install -y fuse-devel

RUN useradd podman; \ echo podman:10000:5000 > /etc/subuid; \ echo podman:10000:5000 > /etc/subgid;

VOLUME /var/lib/containers

RUN mkdir -p /home/podman/.local/share/containers

ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /home/podman/.config/containers/containers.conf

chmod containers.conf and adjust storage.conf to enable Fuse storage.

RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage./a "/var/lib/shared",' -e 's|^mountopt[[:space:]]=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN yum install -y \ python3-pip \ python3 python3-wheel

RUN pip install jupyterlab

COPY login-script.sh /etc/containers/ RUN chmod -R 777 /etc/containers/login-script.sh RUN chown podman:podman -R /home/podman/ USER podman

VOLUME /home/podman/.local/share/containers WORKDIR /data ENTRYPOINT ["/etc/containers/login-script.sh"]

-------------------------------------------------Dockerfile end-------------------------------------------------

sachinkaushik commented 3 years ago

Hi @rhatdan ,

We have deployed Container Image(that we created using above dockerfile) to OpenShift. This container is running as a rootless OpenShift container.

Do we need more previledge for that podman User in OpenShift?

Because when we are building container image in openshift container, we are getting below error.

: using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first fuse-overlayfs: cannot mount: No such file or directory

Can you please help here what else extra config required for same.?

rhatdan commented 3 years ago

Yes, we published an article today that talks about running podman inside of kubernets.

https://www.redhat.com/sysadmin/podman-inside-kubernetes

See if this helps.

rhatdan commented 3 years ago

We have just published

https://www.redhat.com/sysadmin/podman-inside-container And https://www.redhat.com/sysadmin/podman-inside-kubernetes

Please read these and see if they help solve your problem. Reopen if you need more information.

sachinkaushik commented 3 years ago

Hi @rhatdan ,

Thank you..!! Above error has been resolved after following below link. We are creating a rootless openshift container.

I followed Rootless Podman without the privileged flag article.

https://www.redhat.com/sysadmin/podman-inside-kubernetes

But now we are getting below error. Im trying to build python application using dockefile. While Im able to build java spring application. Only problem with python application when there is pip install -r requirements.txt in dockerfile.

  1. If there is a "pip install" command in a Dockerfile, then Podman build fails with error " mount /proc to /proc: Operation not permitted"
  2. Podman build creates docker image, if Dockerfile does not have "pip install" command

Source Code Link : https://github.com/sachinkaushik/hello-world-python.git (Dockerfile has pip install -r requirements.txt)

Command Used : podman build -t python .

error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: mount /proc to /proc: Operation not permitted : exit status 1 Error: error building at STEP "RUN pip install -r requirements.txt": error while running runtime: exit status 1

podman info:

host: arch: amd64 buildahVersion: 1.21.0 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 12 distribution: distribution: fedora version: "34" eventLogger: file hostname: cliservice-7dff79cbd7-7qkt5 idMappings: gidmap:

-----------------------------------------------------------------------Dockerfile------------------------------------------

FROM quay.io/podman/stable:latest

RUN touch /etc/subgid /etc/subuid \ && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ && echo podman:10000:65536 > /etc/subuid \ && echo podman:10000:65536 > /etc/subgid

Use chroot since the default runc does not work when running rootless

RUN echo "export BUILDAH_ISOLATION=chroot" >> /home/podman/.bashrc

RUN yum install -y \ python3-pip \ python3 python3-wheel \ git \ java-11-openjdk.x86_64

RUN pip install jupyterlab

ARG MAVEN_VERSION=3.8.1 ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries

RUN mkdir -p /usr/share/maven /usr/share/maven/ref \ && curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \ && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \ && rm -f /tmp/apache-maven.tar.gz \ && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn \ && yum install wget -y \ && yum install unzip -y \ && wget -q https://services.gradle.org/distributions/gradle-3.3-bin.zip \ && unzip gradle-3.3-bin.zip -d /opt \ && rm gradle-3.3-bin.zip

ENV JAVA_HOME /usr/lib/jvm/jre-11-openjdk/ ENV MAVEN_HOME /usr/share/maven ENV GRADLE_HOME /opt/gradle-3.3 ENV PATH $PATH:/opt/gradle-3.3/bin

COPY registries.conf /etc/containers/ COPY login-script.sh /etc/containers/ RUN chmod -R 777 /etc/containers/login-script.sh

USER podman

WORKDIR /data

ENTRYPOINT ["/etc/containers/login-script.sh"]

rhatdan commented 3 years ago

Please open a new issue. @umohnani8 PTAL

sachinkaushik commented 3 years ago

Hi @rhatdan

I have already open below new issue.

https://github.com/containers/podman/issues/10864

Thanks, Sachin