search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.52k stars 424 forks source link

pull permissions errors #977

Closed vsoch closed 6 years ago

vsoch commented 6 years ago

The Singularity pull command is no longer working, there are weird permissions errors. Likely it didn't get updated with build, or uses a component of it that no longer works?

Version of Singularity:

singularity --version 2.3.9-development.g3dafa39

singularity pull docker://ubuntu /.ubuntu.img

singularity pull docker://ubuntuCreating empty 226MiB image file: ./ubuntu.img
Formatting image with ext3 file system
Image is done: ./ubuntu.img
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/vanessa/.singularity/docker
[5/5] |===================================| 100.0% 
Importing: base Singularity environment
tar: ./etc: Cannot mkdir: Permission denied
tar: ./etc: Cannot mkdir: Permission denied
tar: ./etc/hosts: Cannot open: No such file or directory
tar: ./etc: Cannot mkdir: Permission denied
tar: ./etc/resolv.conf: Cannot open: No such file or directory
tar: ./environment: Cannot create symlink to ‘.singularity.d/env/90-environment.sh’: Permission denied
tar: ./.shell: Cannot create symlink to ‘.singularity.d/actions/shell’: Permission denied
tar: ./dev: Cannot mkdir: Permission denied
tar: ./sys: Cannot mkdir: Permission denied
tar: ./.test: Cannot create symlink to ‘.singularity.d/actions/test’: Permission denied
tar: ./.exec: Cannot create symlink to ‘.singularity.d/actions/exec’: Permission denied
tar: ./root: Cannot mkdir: Permission denied
tar: ./.run: Cannot create symlink to ‘.singularity.d/actions/run’: Permission denied
tar: ./var: Cannot mkdir: Permission denied
tar: ./var: Cannot mkdir: Permission denied
tar: ./var/tmp: Cannot mkdir: No such file or directory
tar: ./proc: Cannot mkdir: Permission denied
tar: ./tmp: Cannot mkdir: Permission denied
tar: ./singularity: Cannot create symlink to ‘.singularity.d/runscript’: Permission denied
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/env: Cannot mkdir: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/env/95-apps.sh: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/env/90-environment.sh: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/env/01-base.sh: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/env/99-base.sh: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/actions: Cannot mkdir: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/actions/test: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/actions/exec: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/actions/shell: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/actions/run: Cannot open: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/libs: Cannot mkdir: No such file or directory
tar: ./.singularity.d: Cannot mkdir: Permission denied
tar: ./.singularity.d/runscript: Cannot open: No such file or directory
tar: ./home: Cannot mkdir: Permission denied
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors
GodloveD commented 6 years ago

Here is the problem. The original image contains root owned files and directories. tar cannot copy these files to the new image with the correct permissions if it is executed as a normal user.

In 2.3.x this worked properly because we did priv escalation on import. But in 2.4 we are stepping back from priv escalation wherever possible to reduce attack surface. (Remember 2.3.1 security update?) So you are getting warnings telling you files are missing or don’t have correct permissions and your container may or may not work afterward. Solution is to do pull as root.

We have the same issue in some of the build routines that essentially do the same thing. There we issue warnings telling the user their container may not function as expected unless they build as root. Should probably do the same thing here as well.

cclerget commented 6 years ago

Leave this one here: https://github.com/singularityware/singularity/issues/963#issuecomment-332162419, it's related to import/export which are not setuid anymore.

@vsoch Don't see a solution without setuid or capabilities ...

vsoch commented 6 years ago

This is a non trivial - this is removing all ability to pull / import as non root. Do the users know about this?

GodloveD commented 6 years ago

OK. After digging into this deeper I found the following.

  1. pull and build are now so similar that it makes sense to deprecate the pull command. At least for Docker images.
  2. build works in a slightly different way than pull. Instead of streaming a bunch of tar pipes into an image, it unpacks them to a temporary directory. Then uses mksqaushfs to convert the directory to an image. This method results in fewer permission errors.
  3. Using pull with Singularity Hub is still just fine because in that case you are just downloading a file. No format conversion needs to occur.
  4. Using build with Singularity Hub as non-root does result in permission errors. That is because the image must be exported to a temporary directory to be converted to squashfs format. This is unavoidable. But the problem should go away as Singularity Hub begins hosting images in squashfs format. At that point a build will also simply be a download.

So after this change, users should be able to do the following without root:

The will still need root to:

But that problem will disappear when Singularity Hub builders are updated and Singularity Hub begins hosting squashfs images.

vsoch commented 6 years ago

Images from singularity hub with pull should not be converted to any different format, they should be pulled (and extracted by python) and then available for use. If a user built an image with squashfs, it will be squashfs. If not, it will be something else.

I hadn't ever tested build from shub, so I can't comment, other than to say you will get back some kind of image, and if build (the new bootstrap) always requires root, it makes sense to need root.

GodloveD commented 6 years ago

@vsoch. That is correct. 😸 But note that build does not always require root. See above.

vsoch commented 6 years ago

And I'll re-iterate here that we cannot, definitely not, make this production until an alternative is provided. My original plan for singularity hub was to update with squash fs AFTER the release, but I think now for the users we need to do:

otherwise we are going to have a break in time when user's cannot achieve the basic goal of getting an image from a registry without root.

GodloveD commented 6 years ago

@vsoch an alternative has been provided. User's should be able to proceed as normal. Crisis averted. Unless I am missing something. If I am, please tell me what user's cannot do.

pditommaso commented 6 years ago

Thanks for continuing to support pull without sudo. This has been a major improvement for many users, because it allows the seamless interoperability with the huge collection of existing docker images (eg. bio-containers, dockstore, etc).

vsoch commented 6 years ago

what is the alternative?

GodloveD commented 6 years ago 
$ singularity --version
2.3.9-development.ge99a6db

$ singularity pull docker://ubuntu
WARNING: Using pull for Docker Hub images is deprecated.
WARNING: Use the build command like so:
WARNING: singularity build ./ubuntu.img docker://ubuntu
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/ubuntu/.singularity/docker
Importing: base Singularity environment
Importing: /home/ubuntu/.singularity/docker/sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118.tar.gz
Importing: /home/ubuntu/.singularity/docker/sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a.tar.gz
Importing: /home/ubuntu/.singularity/docker/sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2.tar.gz
Importing: /home/ubuntu/.singularity/docker/sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e.tar.gz
Importing: /home/ubuntu/.singularity/docker/sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9.tar.gz
Importing: /home/ubuntu/.singularity/metadata/sha256:22e289880847a9a2f32c62c237d2f7e3f4eae7259bf1d5c7ec7ffa19c1a483c8.tar.gz
Building image from sandbox: /tmp/.singularity-build.74XBNe
Building Singularity image...
Cleaning up...
Singularity container built: ./ubuntu.img
vsoch commented 6 years ago

But this part:

WARNING: Using pull for Docker Hub images is deprecated.
WARNING: Use the build command like so:
WARNING: singularity build ./ubuntu.img docker://ubuntu

doesn't that say "this is going away, sorry?"

pditommaso commented 6 years ago

Oh, pull is deprecated, that will break exist applications relying on it :/

GodloveD commented 6 years ago

It still works. It's just warning you that it is deprecated and may not work in a future release. Under the hood it is now using build. So it is just telling you that you should switch your usage to build as well.

bauerm97 commented 6 years ago

@pditommaso that should say legacy instead of deprecated. The command singularity pull docker://image will continue to work as usual. However, one can get the same results using the singularity build image.img docker://image and since we are trying to transition to a unified singularity build command, we want to advertise that instead. It's simply a syntax change, however applications that are using the legacy pull syntax won't break as a result of this change.

vsoch commented 6 years ago

more stupid questions from the peanut gallery - we talk about build with shub root / non root, but we haven't addressed build with non root with docker hub. For example:

build works in a slightly different way than pull. Instead of streaming a bunch of tar pipes into an image, it unpacks them to a temporary directory. Then uses mksqaushfs to convert the directory to an image. This method results in fewer permission errors.

If it needs mksquashfs how is this supported for non root?

vsoch commented 6 years ago

huge +1 on the choice of wording @bauerm97 ! These subtle details are important.

pditommaso commented 6 years ago

Frankly I find pull much more idiomatic than build to download a remote image (from a user perspective) that would justify to not demote it to a legacy command. But yes, it's just an aesthetic aspect.

satra commented 6 years ago

@GodloveD - just to clarify, does build from a docker image also take care of environment variables, runscripts, etc.,. i.e. similar to what the previous pull did?

GodloveD commented 6 years ago

@satra yes. If it doesn't it's a bug.

gmkurtzer commented 6 years ago

The pull command implies that there is an image that has been created, and you are simply retrieving that image, with no modifications so it can be used locally. This is exactly how pull works with Singualrity Hub. Checksums, signatures (as we are working on now) would all be maintained.

With Docker, it is an entirely different situation because we can not use these containers as they exist from Docker Hub. We must convert (or rather build them) into a Singularity compatible container format. That is why the Singularity verb pull does not make sense for Docker containers. Pedantically, if we want to support Docker containers with pull, we should return with a bunch of tarballs unmodified from Docker Hub; this would obviously be rather pointless from a usage perspective of Singularity.

With that said, we can do things to alleviate some of the confusion. I like @GodloveD's suggestion because pull still works because it in turn exec's build behind the scene, I don't like it because it is not in alignment with the intention of pull (however I concede that legacy support is important).

Printing a warning or notice to the user is also important and as @bauerm97 mentioned, that wording can be fine tuned as to proactively mitigate "knee jerk" negative reactions. ;)

gmkurtzer commented 6 years ago

Works for me using build:

$ singularity build ubuntu docker://ubuntu
Building into existing container: ubuntu
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/gmk/.singularity/docker
Importing: base Singularity environment
Importing: /home/gmk/.singularity/docker/sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118.tar.gz
Importing: /home/gmk/.singularity/docker/sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a.tar.gz
Importing: /home/gmk/.singularity/docker/sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2.tar.gz
Importing: /home/gmk/.singularity/docker/sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e.tar.gz
Importing: /home/gmk/.singularity/docker/sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9.tar.gz
Importing: /home/gmk/.singularity/metadata/sha256:22e289880847a9a2f32c62c237d2f7e3f4eae7259bf1d5c7ec7ffa19c1a483c8.tar.gz
Building image from sandbox: /tmp/.singularity-build.IBBRiL
Building Singularity image...
Cleaning up...
Singularity container built: ubuntu
gmkurtzer commented 6 years ago

I believe this has been resolved now. singularity pull docker://.... will now re-exec singularity build ... with a warning about reproducibility. Closing this issue.

jakirkham commented 6 years ago

So I'm having trouble getting this to work with a personal image. Should I open a new issue or are we ok to discuss it here?

vsoch commented 6 years ago

We can re-open it here! It will preserve the previous discussion, if it's needed.

jakirkham commented 6 years ago

Ok. Sure.

Should preface this with I'm very green with Singularity (just starting today), but have a fair bit of experience with Docker.

I tried to run singularity pull docker://jakirkham/centos_drmaa_conda on our cluster, but ran into a bunch of Permission denied errors for things that live in /usr/bin and similar. Almost all of which were things coming from the centos:6 Docker image. Also tried using singularity build centos_drmaa_conda.simg docker://jakirkham/centos_drmaa_conda (as there were some notes about singularity pull being deprecated above), but ran into the same problem. Should add that I had no issues running singularity pull docker://centos:6, which is the base of my Docker image.

As I saw there was a converter tool using Docker, docker2singularity, figured I'd give that a spin locally. Using the command below, was able to convert the image. That seemed to create the image fine. After some slow copying, managed to get the image onto the cluster and run the shell in the image without issues. While this worked, this is a slower workflow than I would like.

docker run -v /var/run/docker.sock:/var/run/docker.sock -v "$(pwd)":/output --privileged -t --rm singularityware/docker2singularity jakirkham/centos_drmaa_conda

So am now left wondering why I wasn't able to get the image on the cluster directly. Any ideas? Should add am using version 2.4.2. Not sure how to get more info out of singularity, but am happy to provide it if you tell me how. :)


Edit: Should add that I can run into the same issues with the much simpler image, jakirkham/centos_conda, also on Docker Hub.

Edit 2: Had similar issues on an Ubuntu machine that I have sudo permissions for (though sudo shouldn't be needed?). This includes trying to convert the image. Only am able to convert on my macOS machine using Docker for Mac.

jakirkham commented 6 years ago

Any thoughts on this? Requests for more information? Would love to be able to pull Docker images into Singularity images directly on our cluster if possible.

satra commented 6 years ago

@jakirkham - you should be able to do this (subject to aufs issues being handled in a separate PR) using, for example, singularity build busybox.img docker://busybox (requires singularity 2.4.x)

jakirkham commented 6 years ago

Sure. As noted above, was able to run singularity pull docker://centos:6, but can't seem to run singularity pull docker://jakirkham/centos_conda, which basically just includes some Miniconda installs on top of centos:6. Are you able to get this to work?

vsoch commented 6 years ago

This image is actually a FROM from another image on quay.io --> https://hub.docker.com/r/jakirkham/centos_conda/~/dockerfile/

So it would be more direct to specify the registry as quay.io. And then there is then an error with applying the container whiteouts, likely a permissions issue too:

singularity pull docker://quay.io/jakirkham/centos_conda
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: quay.io/jakirkham/centos_conda:latest
Cache folder set to /home/vanessa/.singularity/docker
[6/6] |===================================| 100.0% 
Importing: base Singularity environment
ERROR  : Error applying layer whiteouts
ABORT  : Retval = 255

@dctrud any insight off the top of your head about what could be the permissions issue, given the mirror?

jakirkham commented 6 years ago

Yeah Docker Hub has been painful at times. So have mirrored images from Quay in some cases. Sorry if that caused confusion. Probably will replace both with just pushing from CircleCI at some point.

dtrudg commented 6 years ago

@vsoch @jakirkham - PR #1261 may fix part of this. I then get the container running of sorts:

[WARN  tini (16048)] Tini is not running as PID 1 .
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, run Tini as PID 1.

... but during extraction I'm seeing a lot of warnings from libarchive (which is handling the tar files) about unable to create things. Haven't ever seen these from any other container, so will need some digging.

jakirkham commented 6 years ago

So if we think it is an issue with whiteouts, we could try squashing the image. Tried building the image locally with the --squash flag and pushed it to Docker Hub as jakirkham/centos_conda:squashed. Can verify that the history (except for the centos:6 part) is squashed into one layer. Tried pulling this onto the cluster via singularity pull docker://jakirkham/centos_conda:squashed. However it still fails with a variety of Permission denied errors on different files.

jakirkham commented 6 years ago

Thanks for playing with this a bit @dctrud.

So tini is a subprocess reaper commonly used in Docker ( particularly before the advent of --init for docker run, which also uses tini ;) ). This warning can be ignored. Probably can even disable tini with Singularity somehow (its part of the Docker ENTRYPOINT).

What sort of warnings is libarchive giving?

dtrudg commented 6 years ago

@jakirkham - is this on dev or with PR #1261 ? I think there are 2 issues here. PR #1261 fixes something that is almost certainly tripping you up with the non-squashed image.

With PR#1261 I get the same thing on either non-squashed, or squashed - a bunch of:

WARNING: Warning handling tar header: Can't unlink already-existing object
WARNING: Warning handling tar header: Can't create .....

...but tini starts.

dtrudg commented 6 years ago

Okay - found something. Manually extracting the first of the squashed container layers, various important directories in /usr are ending up with no 'w' permission set for anyone:

From an ls usr/ of extracted first layer:

dr-xr-xr-x  8 dave dave   98 Jan  7 11:03 lib
dr-xr-xr-x 21 dave dave 8.0K Jan 30 14:28 lib64

What constitutes the first layer of this squashed image? What's the original FROM, and what's being done to it in the Dockerfile?

I guess we can force a +rw perms for the owner throughout the extracted tree at each layer step, but I'd like to know how we end up with a tar that extracts usr/lib etc. without owner write perms

jakirkham commented 6 years ago

The pull was with Singularity 2.4.2.

jakirkham commented 6 years ago

The FROM is centos:6. Here’s the Dockerfile for more details. The bulk of it is just installing two different copies of Miniconda.

vsoch commented 6 years ago

This looks familiar to something we had a long time ago (when singularity was new) - some extracted files wouldn't have any permission, and we did a search for them to fix permissions during download. If we jump over the mirror Dockerfile on Docker Hub and grab quay directly, I can see the empty layers:

SINGULARITY_FIX_PERMS=true
export SINGULARITY_FIX_PERMS
 singularity pull docker://quay.io/jakirkham/centos_conda
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: quay.io/jakirkham/centos_conda:latest
Cache folder set to /home/vanessa/.singularity/docker
[1/12] |==|--------------------------------|   8.3% WARNING Tar file /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.Z01Dm_ is empty, skipping.
[3/12] |========|--------------------------|  25.0% WARNING Tar file /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.golYbn is empty, skipping.
[5/12] |==============|--------------------|  41.7% WARNING Tar file /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.eBMzBL is empty, skipping.
[7/12] |====================|--------------|  58.3% WARNING Tar file /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.vL18KM is empty, skipping.
[9/12] |==========================|--------|  75.0% WARNING Tar file /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.zuMT35 is empty, skipping.
[11/12] |================================|--|  91.7% WARNING Tar file /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.u7lv7G is empty, skipping.
[12/12] |===================================| 100.0% 
Importing: base Singularity environment
ERROR  : Error applying layer whiteouts
ABORT  : Retval = 255
jakirkham commented 6 years ago

Thanks for the follow-up, @vsoch.

Interesting. Any ideas why some files wouldn't have permissions? Anything I can read up on this?

There are some empty layers due to things like LABEL, CMD, MAINTAINER, ENV, ENTRYPOINT, etc. Counted 6 in all (2 uses of CMD and 1 of everything else). So this matches what one might expect with the warnings above.

vsoch commented 6 years ago

I'm not sure, I remember conversation with @gmkurtzer probably about a year ago about finding and changing permissions for files that didn't have permissions. @gmkurtzer do you know why this would happen?

The empty layers make sense - that indeed is what happens with the "metadata labels" like LABEL, CMD, ENV, etc.

fearful-symmetry commented 6 years ago

I seem to be running into a similar issue with singularity -vv build es_test.img docker://docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.3 

tar: usr/bin/abs2rel: Cannot open: Permission denied
tar: usr/bin/addr2line: Cannot open: File exists
tar: usr/bin/ar: Cannot open: File exists
tar: usr/bin/as: Cannot open: File exists
tar: usr/bin/bootctl: Cannot open: File exists
tar: usr/bin/build-classpath: Cannot open: Permission denied
tar: usr/bin/build-classpath-directory: Cannot open: Permission denied
tar: usr/bin/build-jar-repository: Cannot open: Permission denied
tar: usr/bin/busctl: Cannot open: File exists
tar: usr/bin/c++filt: Cannot open: File exists
jakirkham commented 6 years ago

Friendly nudge. 😉 Any ideas on a workaround for this issue?

dtrudg commented 6 years ago

Hi @jakirkham, @fearful-symmetry - I'm looking at a fix for this on non-root builds at the moment. Will post more here soon

jakirkham commented 6 years ago

Great, thanks @dctrud! Please let me know if there is any more information I can provide or if there is something that I should test. 😄

vsoch commented 6 years ago

+1 @dctrud and I'd like to add a fix to sregistry too

dtrudg commented 6 years ago

Hi all - if you are able to test PR #1295 that would be great. This builds on other work to fix whiteout/permissions issues with docker layer extraction, and addresses the centos_conda and elasticsearch extraction problems above

dtrudg commented 6 years ago

Closing this as PRs fixing whiteout/perms issues have been merged to development. Let us know if there are any remaining problems. Thanks!