search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.98k stars 2.34k forks source link

error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: mount `/proc` to `/proc`: Operation not permitted #10864

Closed sachinkaushik closed 2 years ago

sachinkaushik commented 3 years ago

Hi Team,

I have created a running rootless openshift container using a Dockerfile. I followed below link for creating Rootless Podman without the privileged flag. I'm able to build java spring application but when I try to build python application using Dockerfile that has pip install then I'm getting below error. Can you please let us know what else config required to resolve below error?

https://www.redhat.com/sysadmin/podman-inside-kubernetes

error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: mount /proc to /proc: Operation not permitted

    • If there is a "pip install" command in a Dockerfile, then Podman build fails with error " mount /proc to /proc: Operation not permitted"
    • Podman build creates docker image, if Dockerfile does not have "pip install" command

podman --version :: podman version 3.2.2

podman info ::

host: arch: amd64 buildahVersion: 1.21.0 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 12 distribution: distribution: fedora version: "34" eventLogger: file hostname: cliservice-7dff79cbd7-n7krd idMappings: gidmap:

------------------------------------------------------Dockerfile- Start-------------------------------------------

FROM quay.io/podman/stable:latest

RUN touch /etc/subgid /etc/subuid \ && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ && echo podman:10000:5000 > /etc/subuid \ && echo podman:10000:5000 > /etc/subgid

RUN yum install -y \ python3-pip \ python3 python3-wheel \ git \ java-11-openjdk.x86_64

RUN pip install jupyterlab

ARG MAVEN_VERSION=3.8.1 ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries

RUN mkdir -p /usr/share/maven /usr/share/maven/ref \ && curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \ && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \ && rm -f /tmp/apache-maven.tar.gz \ && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn \ && yum install wget -y \ && yum install unzip -y \ && wget -q https://services.gradle.org/distributions/gradle-3.3-bin.zip \ && unzip gradle-3.3-bin.zip -d /opt \ && rm gradle-3.3-bin.zip

ENV JAVA_HOME /usr/lib/jvm/jre-11-openjdk/ ENV MAVEN_HOME /usr/share/maven ENV GRADLE_HOME /opt/gradle-3.3 ENV PATH $PATH:/opt/gradle-3.3/bin

COPY registries.conf /etc/containers/ COPY login-script.sh /etc/containers/ RUN chmod -R 777 /etc/containers/login-script.sh USER podman

WORKDIR /data

ENTRYPOINT ["/etc/containers/login-script.sh"]

-------------------------------------------Dockerfile End-------------------------------------------

podman - proc

sachinkaushik commented 3 years ago

Hi Team,

Any update on this...?

baude commented 3 years ago

@rhatdan @umohnani8 ptal

flouthoc commented 3 years ago

Hi @sachinkaushik is this rootless container being invoked from another rootless/non-root container ? Could you try adding this to your podman command --security-opt seccomp=unconfined --cap-add all ?

flouthoc commented 3 years ago

also afaik parent container has to be privileged and must mount parts of /proc with relevant uid,gid for nested rootless container to be able to perform mount on procfs i am not sure about it though. @sachinkaushik Could you please try with privileged: true if above suggested methods don't work.

sachinkaushik commented 3 years ago

HI @flouthoc ,

Thank you for response..!!!

We have created a Container Image using below Dockerfile by docker build -t . command. And now this container image we are running as rootless(we have podman User in dockerfile) container in Openshift.

This is a rootless container running in openshift. Now if we try to build python application Dockerfile that is having pip install command then only we are getting error mentioned in issue subject.

Using below Dockerfile we have created Container Image and same we have deployed in Openshift and that is running as rootless container and inside this we are trying build python application.

------------------------------------Dockerfile start------------------------------------------------------

FROM quay.io/podman/stable:latest

RUN touch /etc/subgid /etc/subuid \ && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ && echo podman:10000:5000 > /etc/subuid \ && echo podman:10000:5000 > /etc/subgid

RUN yum install -y \ python3-pip \ python3 python3-wheel

RUN pip install jupyterlab

ENV PATH $PATH:/opt/gradle-3.3/bin

COPY registries.conf /etc/containers/ COPY login-script.sh /etc/containers/ RUN chmod -R 777 /etc/containers/login-script.sh USER podman

WORKDIR /data

ENTRYPOINT ["/etc/containers/login-script.sh"]

----------------------------------------------Dockerfile end------------------------------------------------------

Note : We have to give less privileged to User.

flouthoc commented 3 years ago

@sachinkaushik oh its fine if you don't want to try privileged: true but could you try this: podman build --security-opt seccomp=unconfined --cap-add all -t <image-name> . and tell me the output ?

sachinkaushik commented 3 years ago

Hi @flouthoc ,

I just try it and getitng same error.

podman build --security-opt seccomp=unconfined --cap-add all -t python-image .

STEP 5: RUN pip install -r requirements.txt error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: mount /proc to /proc: Operation not permitted

podman-build-error

I followed Rootless Podman without the privileged flag article.

https://www.redhat.com/sysadmin/podman-inside-kubernetes

flouthoc commented 3 years ago

@sachinkaushik and just for a try what happens when you set privileged: true on the pod config ?

sachinkaushik commented 3 years ago

@flouthoc We have created s SCC and in that we have allowPrivilegedContainer: false . Do you want us to set value of allowPrivilegedContainer as true ?

sachinkaushik commented 3 years ago

@flouthoc We tried setting up value of allowPrivilegedContainer as true. But still no luck.

STEP 5: RUN pip install -r requirements.txt error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: mount /proc to /proc: Operation not permitted

sachinkaushik commented 3 years ago

HI @rhatdan , @flouthoc ,

Any update on this?

flouthoc commented 3 years ago

@sachinkaushik I was not able to spend time on this yesterday will probably re-create this on my end and will try a few things. btw when you tried allowPrivilegedContainer: true did you update your defined SCC as well ?

sachinkaushik commented 3 years ago

Hi @flouthoc ,

Yes we updated.

We are using Service Account and that SA is bind with Role. And that role is having a below SCC.

------------------------------------------------------------SCC Start------------------------------------- allowHostPorts: false priority: 10 requiredDropCapabilities:

rhatdan commented 3 years ago

The issue is the outer container has setup /proc with certain read/only mounts and mounted over parts of /proc, When running podman container inside it tries to modify /proc mount and the kernel does not allow this. So you can either do an --unmask=/proc/* or --unmask=all on the outside container. or volume mount -v /proc:/proc on the inside container. (I believe).

@giuseppe WDYT?

sachinkaushik commented 3 years ago

HI @rhatdan / @flouthoc ,

I tried above things but still no luck...

error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: mount /proc to /proc: Operation not permitted

I think there is problem with crun, it doesn't have permission to mount proc.

We have below dockerfile and trying to build container image of it. But step 5 <RUN pip install -r requirements.txt> gives error.

-----------------------Docker file--------------------

FROM python:3-alpine MAINTAINER Sachin Sharma WORKDIR /service COPY requirements.txt . RUN pip install -r requirements.txt COPY . ./ EXPOSE 8080 ENTRYPOINT ["python3", "app.py",]

flouthoc commented 3 years ago

@sachinkaushik I tried recreating your use-case podman build -t img-python . with one of your repos https://github.com/sachinkaushik/hello-world-python.git inside a rootful privileged podman container started using sudo podman run --privileged quay.io/podman/stable sleep 100000000 but everything worked completely fine for me. Sharing the complete output of build inside a container https://paste.ubuntu.com/p/c4Mh99dScd/

I did this by-the-way.

Podman version: 3.3.0-dev Crun version: 0.20.1.17-0b0b

flouthoc commented 3 years ago

Same case for rootless privileged container started using podman run --privileged quay.io/podman/stable sleep 100000000. I am unable to reproduce this case everything is working just fine for me.

Steps i did

sachinkaushik commented 3 years ago

HI @flouthoc / @rhatdan ,

This is working as rootful container our end also. But when Im running as rootless container then getting below error. This is new error now. We have priviledge true as well in Pod YAML file. Please help me here what else config I'm missing to add.

securityContext: privileged: true

Error :

STEP 5: RUN pip install -r requirements.txt error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: newgidmap: gid range [0-4294967295) -> [0-4294967295) not allowed writing file /proc/248/gid_map: Invalid argument

Im creating container using below docker file.

FROM quay.io/podman/stable:latest

RUN yum install -y \ python3-pip \ python3 python3-wheel \ git \ java-11-openjdk.x86_64

RUN pip install jupyterlab

USER podman

WORKDIR /data

ENTRYPOINT ["jupyter", "lab", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

Working with root User.

podman-rrot

umohnani8 commented 3 years ago

@sachinkaushik have you tried the build with the --isolation=chroot flag as the article says? That should fix the permisison denied you are getting for mounting /proc. The chroot isolation helps bypass some of the things that the kernel denies mounting in the default isolation setting. So far builds in a rootless unprivileged container only works with --isolation=chroot.

Another option would be to run your pod in a user namespace using the cri-o runtime class method. The steps for doing this is in the article as well.

Article: https://www.redhat.com/sysadmin/podman-inside-kubernetes

sachinkaushik commented 3 years ago

@umohnani8 Now im not getting proc mount error. Now there is below error when I'm running as rootless container. I have added subuid and subguid in dockerfile as well, as mentioned in article.

error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: newgidmap: gid range [0-4294967295) -> [0-4294967295) not allowed writing file /proc/234/gid_map: Invalid argument : exit status 1

Error

And if try with--isolation=chroot flag, using below command getting another error.

podman build --isolation chroot -t demo

STEP 5: RUN pip install -r requirements.txt error setting capabilities for process: error reading capabilities of current process: open /proc/self/status: permission denied subprocess exited with status 1

Dockerfile :

FROM quay.io/podman/stable:latest

RUN echo umohnani:100000:65536 > /etc/subuid; \ echo containers:200000:268435456 > /etc/subuid; \ echo umohnani:100000:65536 > /etc/subgid; \ echo containers:200000:268435456 > /etc/subgid;

RUN yum install -y \ python3-pip \ python3 python3-wheel \ git \ java-11-openjdk.x86_64

RUN pip install jupyterlab

COPY login-script.sh /etc/containers/ RUN chmod -R 777 /etc/containers/login-script.sh

USER podman

WORKDIR /data

ENTRYPOINT ["jupyter", "lab", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

jupyter lab --port=8888 --no-browser --ip=0.0.0.0 --allow-root

flouthoc commented 3 years ago

@sachinkaushik Could you please remove the lines from you Containerfile/Dockerfile

RUN echo umohnani:100000:65536 > /etc/subuid;
echo containers:200000:268435456 > /etc/subuid;
echo umohnani:100000:65536 > /etc/subgid;
echo containers:200000:268435456 > /etc/subgid;

and try

sachinkaushik commented 3 years ago

@flouthoc first I tried without above lines from dockerfile for rootless container. I was getting below error. Then I tried with above lines but still there was same error.

error running container: error from /usr/bin/crun creating container for [/bin/sh -c pip install -r requirements.txt]: newgidmap: gid range [0-4294967295) -> [0-4294967295) not allowed writing file /proc/234/gid_map: Invalid argument

flouthoc commented 3 years ago

Could you please share idmappings of your podman, you can get that by podman info

sachinkaushik commented 3 years ago

@flouthoc Please find below output of podman info

podman info ::

host: arch: amd64 buildahVersion: 1.21.0 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 12 distribution: distribution: fedora version: "34" eventLogger: file hostname: cliservice-874f8bb78-rt7wt idMappings: gidmap:

I created container using below Dockerfile.

FROM quay.io/podman/stable:latest

RUN yum install -y \ python3-pip \ python3 python3-wheel \ git \ java-11-openjdk.x86_64

RUN pip install jupyterlab

USER podman

WORKDIR /data

ENTRYPOINT ["jupyter", "lab", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

flouthoc commented 3 years ago

@sachinkaushik try a smaller range and also add entry for root by adding this to containerfile

RUN touch /etc/subgid /etc/subuid
&& chmod g=u /etc/subgid /etc/subuid /etc/passwd
&& echo root:165536:65536 > /etc/subuid
&& echo root:165536:65536 > /etc/subgid
&& echo containers:165536:65536 > /etc/subgid
&& echo containers:165536:65536 > /etc/subuid
&& echo podman:10000:5000 > /etc/subuid
&& echo podman:10000:5000 > /etc/subgid
sachinkaushik commented 3 years ago

@flouthoc I tried after adding above lines in Dockerfile but still same error.

FROM quay.io/podman/stable:latest

RUN touch /etc/subgid /etc/subuid \ && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ && echo root:165536:65536 > /etc/subuid \ && echo root:165536:65536 > /etc/subgid \ && echo containers:165536:65536 > /etc/subgid \ && echo containers:165536:65536 > /etc/subuid \ && echo podman:10000:5000 > /etc/subuid \ && echo podman:10000:5000 > /etc/subgid

RUN yum install -y \ python3-pip \ python3 python3-wheel \ git \ java-11-openjdk.x86_64

RUN pip install jupyterlab

USER podman

WORKDIR /data

ENTRYPOINT ["jupyter", "lab", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

podman info :

[podman@cliservice-874f8bb78-xhfds hello-world-python]$ podman info host: arch: amd64 buildahVersion: 1.21.0 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 12 distribution: distribution: fedora version: "34" eventLogger: file hostname: cliservice-874f8bb78-xhfds idMappings: gidmap:

sachinkaushik commented 3 years ago

@flouthoc @umohnani8 any update on this?

flouthoc commented 3 years ago

@sachinkaushik Could you share contents of /etc/subuid and /etc/subgid from your container.

sachinkaushik commented 3 years ago

@flouthoc Please find below same.

[podman@cliservice-688f5bb747-4d6nb hello-world-python]$ cat /etc/subuid podman:10000:5000 [podman@cliservice-688f5bb747-4d6nb hello-world-python]$ cat /etc/subgid podman:10000:5000

subui - subgid

flouthoc commented 3 years ago

@sachinkaushik does your deployment/pod spec contains ? Although i am unsure if this is going to help or not.

     securityContext:
       privileged: true
       runAsUser: 10000
sachinkaushik commented 3 years ago

@flouthoc I have below config in deployment/pod spec. Do you want me add runAsUser: 1000 ?

securityContext: privileged: true

flouthoc commented 3 years ago

runAsUser: 10000 but i am really unsure if this is going to be of any help just give it a try.

sachinkaushik commented 3 years ago

@flouthoc I added in deployment yaml file, and still same error is there.

image

Same Error :

image

flouthoc commented 3 years ago

@sachinkaushik user was supposed to be 10000

sachinkaushik commented 3 years ago

@flouthoc after adding 10000, getting error "cannot find name for user ID 10000"

image

image

flouthoc commented 3 years ago

and what happens with you set runAsUser to 0 ?

sachinkaushik commented 3 years ago

@flouthoc After setting up runAsUser to 0. It just created a rootful container, And with rootful it is working fine already.

image

flouthoc commented 3 years ago

@sachinkaushik okay you can remove runAsUser from spec if you dont want it to be root or runAsUser: 1000 and use following command while performing build from inside of container podman build --security-opt seccomp=unconfined --cap-add all --isolation=chroot -t <image-name> .

sachinkaushik commented 3 years ago

@flouthoc I removed that config. And tried with command you mentioned above. Now there is below error.

error setting capabilities for process: error reading capabilities of current process: open /proc/self/status: permission denied

image

flouthoc commented 3 years ago

@sachinkaushik try with podman build --userns-gid-map-group podman --userns-uid-map-user podman -t <name> . and with podman build --isolation=chroot --userns-gid-map-group podman --userns-uid-map-user podman -t <name> . also could you please share output for both ?

flouthoc commented 3 years ago

@rhatdan @umohnani8 could you guys please let me know if the case is documented or ever tried before for using rootless buildah within a rootless container i think we might have hit a bug not sure.

sachinkaushik commented 3 years ago

@flouthoc I tried with both, please find below output of both commands.

podman build --userns-gid-map-group podman --userns-uid-map-user podman -t img .

image

podman build --isolation=chroot --userns-gid-map-group podman --userns-uid-map-user podman -t img .

image

flouthoc commented 3 years ago

@sachinkaushik I think this should solve try this , this might look weird but give it a shot this should fix your use case. Add this to dockerfile which is being used to start container, please make sure values are exact.

RUN echo test:1000:1 >> /etc/subuid
RUN echo test:1000:5000 >> /etc/subgid

Exec into Pod

Step 1

cat /etc/subuid Should look like

podman:10000:5000
test:1000:1

Step 2

cat /etc/subgid should look like

podman:10000:5000
test:1000:5000

Try running the build using 

podman build --isolation=chroot --userns-gid-map-group test --userns-uid-map-user test -t img .

Eveything should work fine, please use steps 1 and 2 to verify things before running build

sachinkaushik commented 3 years ago

@flouthoc Do I have to use below lines also in containerfile ?

RUN touch /etc/subgid /etc/subuid \ && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ && echo root:165536:65536 > /etc/subuid \ && echo root:165536:65536 > /etc/subgid \ && echo containers:165536:65536 > /etc/subgid \ && echo containers:165536:65536 > /etc/subuid \ && echo podman:10000:5000 > /etc/subuid \ && echo podman:10000:5000 > /etc/subgid

flouthoc commented 3 years ago

@sachinkaushik not needed you can remove all these and add just 

RUN echo test:1000:1 >> /etc/subuid
RUN echo test:1000:5000 >> /etc/subgid
flouthoc commented 3 years ago

@sachinkaushik So i was able to reproduce and these steps worked for me so verify every step before performing the build from https://github.com/containers/podman/issues/10864#issuecomment-879764837 and also add runAsUser:1000 if needed.

sachinkaushik commented 3 years ago

@flouthoc Yes Thank you...!!!,

This is working for now. But Are we going with approach? I mean passing args while we build container image?

Because this feature will be used by end Users. So we don't want to force them to pass all those args while creating container image.

image

flouthoc commented 3 years ago

@sachinkaushik Cheers !!! :tada: :tada: :rocket: , it should unblock you and there should be no problem with using flags but i still think there is bug. Let @rhatdan and @umohnani8 confirm.

sachinkaushik commented 3 years ago

@flouthoc Thank you for help and support..!!! :)

We tried without test user, instead of this we just set range of Ids for podman user only.

&& echo podman:1000:1 > /etc/subuid \ && echo podman:1000:5000 > /etc/subgid

This is also working just with below same command.

podman build --isolation=chroot --userns-gid-map-group podman --userns-uid-map-user test -t podman .

flouthoc commented 3 years ago

@sachinkaushik Cheers !!!! Yes this should work with any user including podman. Lets wait for @umohnani8 and @rhatdan to have a look then we can decide if we want to close issue by settling on this solution or should seek a better approach or mark this as a bug to be fixed.