search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

Same container, different results #476

Closed sysmso closed 7 years ago

sysmso commented 7 years ago

Hi I've created an singularity container for a simple python program on a Centos 7 server. When i run this container on my laptop, i have an error. On other Centos 7 server in the cluster it works well.

Laptop : Linux 4.9.6-1-ARCH #1 SMP PREEMPT Thu Jan 26 09:22:26 CET 2017 x86_64 GNU/Linux Server : Linux 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

On the server : [sysmso@server singularity]$ singularity --version 2.2 [sysmso@server singularity]$ singularity exec ubuntu.img python tenso.py (0, array([ 0.63393396], dtype=float32), array([ 0.02993923], dtype=float32))

On my laptop : [sysmso@laptop singularity]$ singularity --version 2.2 [sysmso@laptop singularity]$ singularity exec ubuntu.img python tenso.py Traceback (most recent call last): File "tenso.py", line 21, in init = tf.global_variables_initializer() AttributeError: 'module' object has no attribute 'global_variables_initializer'

If i understand well the purpose of Singularity i should have the same result on every hosts ? If i'm right, how to debug this ?

Thanks for your help

yarikoptic commented 7 years ago

singularity

  1. mounts your home directory as well -- if you have some Python modules installed under ~/.local etc, your tenso.py might be picking those up, thus making execution dependent on the host if you have those differ among hosts. I guess for complete reproducibility you would need to point to some sanitized $HOME in both cases. smth like (-H available since 2.2 seems to me) 
    rm -fr /tmp/temp_home && mkdir -p /tmp/temp_home; singularity exec -H /tmp/temp_home dbic-bids-env.img  /bin/bash
  2. Singularity passes your environment variables inside -- so if you have those differ on two hosts and something in your script relies on them (PYTHONPATH etc) -- results could differ. I do not see any setting to avoid passing environment variables.

    Would be interesting to know if not passing environment is possible and it better become possible in the long run ;)

sysmso commented 7 years ago

Ok it works with -H, i didn't know this option (not in the man). Thanks for your help

vsoch commented 7 years ago

I'll add this to the main help (and singularity site) - let's keep this issue open.

vsoch commented 7 years ago

ok! I have a PR in to add the -H and --home detail to the --help output, and I've also updated our main docs to have more clear links to the Troubleshooting section, which is hidden under FAQ and was hard to find. Specifically, now the user can find troubleshooting via:

I've also added our recent discussion above here. Let us know if the issue is resolved and you are happy with the docs!

sysmso commented 7 years ago

Yes it's great thank you !

yarikoptic commented 7 years ago

re:

rm -rf /tmp/homie 
mkdir -p /tmp/homie
singularity exec -H /tmp/homie analysis.img /bin/bash

I would strongly recommend to either use '&&' to join all the actions, i.e.

rm -rf /tmp/homie && mkdir -p /tmp/homie && \
singularity exec -H /tmp/homie analysis.img /bin/bash

or use some directory under $HOME, e.g. $HOME/singularity_home. Reasons are numerous, as an example I, as a malicious user, could create /tmp/homie with .bashrc containing just rm -rf $HOME/*. Your first and second command would fail since you wouldn't own that directory, but overall it would get to the 3rd line and then get to run that .bashrc.

edit 1: having such ~/singularity_home might in general be advised as to provide custom "sanitized" environment within singularity containers

vsoch commented 7 years ago

fixed per your suggestion:

image

thanks @yarikoptic !

chrisgorgo commented 7 years ago

I think this situation speaks to my suggestion not to mount $HOME by default (see #445). A few other people also run into this (for example @russpold).

vsoch commented 7 years ago

I think Russ bot :robot: is @poldrack on here

...too many username spaces... cannot compute... beepbeep boooooooooooop /death

truatpasteurdotfr commented 7 years ago

here's my chrome launcher which create a one-time $HOME each time... off course ymmv :P chrome-private.sh.txt

jsmedmar commented 6 years ago

Hey what if the image has the software installed in the container's /home? see https://hub.docker.com/r/ensemblorg/ensembl-vep/~/dockerfile/

vsoch commented 6 years ago

If there is software in the container in home it will be mounted over by the users home unless you add an argument to prevent this (specifying another home with --home) or --containall)

jsmedmar commented 6 years ago

@vsoch thank you using --home helped!

vsoch commented 6 years ago

Glad to help! This issue is kind of like...

image

yarikoptic commented 6 years ago

Given that I had similar gotchas due to /tmp being bind mounted, it seems like might be a good idea to be able to say to not bind mount anything during build. Is it possible?

vsoch commented 6 years ago

Have you tried --isolated?

yarikoptic commented 6 years ago

I guess I didn't. I will from now on ;-)

vsoch commented 6 years ago

It's an awesome feature, @cclerget is amazing!

EvanTheB commented 5 years ago

If there is software in the container in home it will be mounted over by the users home unless you add an argument to prevent this (specifying another home with --home) or --containall)

@vsoch with --containall $HOME is empty, the files put there during build are gone. With -H the files are visible. Is this the intended behaviour?

vsoch commented 5 years ago

A few questions.

  1. Are you sure that the $HOME you intended was defined during build? There was a bug so $HOME and ~ were undefined.
  2. Are you sure it's not in root's home?

And finally, please provide a SIngularity version, recipe, and command to reproduce your case. We cannot easily help without that.

yarikoptic commented 5 years ago

FWIW, here is the helper I came up with to run computation in singularity containers while trying to avoid side-effects of the outside system: https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd

vsoch commented 5 years ago

Also hugely important for PYTHON - doesn't forget to unset PYTHONPATH and PYTHONUSERBASE (that pesky .local folder that everyone forgets about).

EvanTheB commented 5 years ago

@yarikoptic thanks for the script, this looks like it implements the behaviour I expected to be default for singularity!

@vsoch

build machine: ubuntu18.04: 2.4.2-dist run machine: centos7: singularity version 3.0.2-1.el7.centos

Build script:

Bootstrap: docker
From: ubuntu:18.04

%post
    ls $HOME
    mkdir -p /home/evaben/tmp/singularitydir

Build message:

+ ls /root
+ mkdir -p /home/evaben/tmp/singularitydir

Singularity defaults, /home/evaben/tmp/singularitydir is gone: I don't know what the behaviour should be, there is a conflict between 'singularity mounts home' and the file existing. Should they be merged or should the home mount hide the file? I think either behaviour is justifiable. 

> singularity shell test.sif
Singularity test.sif:~/code/test/singularity> pwd
/home/evaben/code/test/singularity
Singularity test.sif:~/code/test/singularity> ls /home/evaben/tmp/singularitydir
ls: cannot access '/home/evaben/tmp/singularitydir': No such file or directory
Singularity test.sif:~/code/test/singularity> exit

-H, /home/evaben/tmp/singularitydir is there, (and my pwd ~/code): This makes sense, and matches what you stated.

> singularity shell -H /tmp/evaben/ test.sif
Singularity test.sif:/home/evaben/code/test/singularity> pwd
/home/evaben/code/test/singularity
Singularity test.sif:/home/evaben/code/test/singularity> ls /home/evaben/tmp/singularitydir
Singularity test.sif:/home/evaben/code/test/singularity> cd
Singularity test.sif:/tmp/evaben> ls
Singularity test.sif:/tmp/evaben> cd /home/evaben/
Singularity test.sif:/home/evaben> ls
code  tmp
Singularity test.sif:/home/evaben> exit

Containall, home dir is empty, /home/evaben/tmp/singularitydir is gone: This does not match what you said. I do not know what the behaviour should be. Does singularity say, -C mounts an EMPTY home directory? Or does it say -C does NOT mount a home directory? Those are different things.

> singularity shell -C test.sif
Singularity test.sif:~> pwd
/home/evaben
Singularity test.sif:~> ls /home/evaben/tmp/singularitydir
ls: cannot access '/home/evaben/tmp/singularitydir': No such file or directory
Singularity test.sif:~> exit
vsoch commented 5 years ago

hey @EvanTheB - most of the the behavior (minus the last bit) you have shown is expected - I started to test this out for you but ran into an issue #3480 so I'll do my best to comment now testing with an already build container.

preparing home...

$ mkdir -p /tmp/home
$ touch /tmp/home/gibbles.txt

and shelling in...

$ singularity shell -H /tmp/home container.sif 
Singularity> echo $HOME
/tmp/home
Singularity> ls $HOME
gibbles.txt
Singularity> ls /home
Singularity>

If you are saying that you are specifying another home, and when you do that you don't see the files in the home that you created during build, this I think is a bug. You should see the files, unless there is some change to Singularity I'm not aware of. If it's the case that your $PWD is being bound, and this conflicts with the changes to home in the container, this could be a reason for not seeing the files. I think this is what you thought of, because your next attempt was to use --containall. Actually, let me try that with my container - shelling in from a $PWD that is in my home:

First confirm that we still have our gibbles home:

$ singularity shell -H /tmp/home /tmp/container.sif 
Singularity> echo $HOME
/tmp/home
Singularity> ls $HOME
gibbles.txt

Now take a look at what's in my actual home

Singularity> ls /home/vanessa/
Documents

Note that I only see one folder (there are many missing) and it's the one that is the base of my $PWD. And here is the full contents of the $PWD, bound by default:

Singularity> ls /home/vanessa/Documents/Dropbox/
Apps         Code         Personal     Research     Resume       Saves        School       Sites        Transcripts  Video        Website

Yep, that's consistent with the theory! I have a lot of other crap in $HOME, but I can only see the $PWD that I'm in (/home/vanessa/Documents/Dropbox)

Okay so now let's see what happens when we add --containall. Technically, containall shouldn't bind anything. But if we tell it to add home, as far as I know, it should add the home.

First, confirm the home I asked for is there:

$ singularity shell --containall -H /tmp/home /tmp/container.sif 
Singularity> echo $HOME
/tmp/home
Singularity> ls $HOME
gibbles.txt

Next confirm that everything else is not there (because we asked for --containall)

Singularity> ls /home/vanessa
ls: /home/vanessa: No such file or directory

This is what I'd expect to. If I remove asking for home, I am indeed going to get a completely isolated environment:

$ singularity shell --containall  /tmp/container.sif 
Singularity> echo $HOME
/home/vanessa
Singularity> ls /home/vanessa/
Singularity> ls /tmp/home
ls: /tmp/home: No such file or directory

So this might be where you have issue? What we see above is that we are still provided with a $HOME, but it's empty. I would hope that if you added files there during build, you would see them (the container I'm testing didn't have files added so it's empty). But from what you've shown, it looks like --containall is also hiding the files that (should be) in the container. I believe this is not expected behavior, and a bug. If others (maintainers) don't see this comment and respond within the working day I would open an issue.

One thing to test in the meantime is what happens if you add home for another user. For example, it could be that the bug is that --containall just empties those locations by default, your actual $HOME being one of them. This isn't a workaround or a fix, but it would shed some light on the behavior that you see. I would also try running the command with --debug to see if there are any messages printed about the decisions to contain / bind, etc. Good luck!

EvanTheB commented 5 years ago

Hi @vsoch, thanks for your investigation. Can I summarise:

Hopefully the behaviour can be changed or the documentation updated to explain why this is the case!

vsoch commented 5 years ago

@EvanTheB I still think it's a bug that --containall doesn't include the files that were originally put in the container. Intuitevely, contain all should imply that I'm isolating (not binding) those locations, so whatever is defined in the container there should persist (and not be wiped out). The case of having a PWD in the path does make sense (and the warning that it was already bound is taken away). Are you going to open an issue?

jmstover commented 5 years ago

If you do not want to mount your home, but do want the current working directory if that is in the $HOME path, use --no-home. The --no-home logic, just doesn't do the automatic mounting of the users home directory, but does still mount the current working directory... which could be your $HOME.

Using --containall, $HOME is not mounted and a dummy bind mount is created at the $HOME point. You cannot use -B to mount your $HOME at this point because it creates an empty mount. So if you have files in the image at /home/user ... --containall will hide that.

If you have a /home/user (where /home/user == $HOME) in the image you want to use, use the --no-home option.

vsoch commented 5 years ago

@jmstover so there is no way to put files in (your) user home during container build, and then have access to them. This seems like a bug - contain-all shouldn't bind the actual home, and also shouldn't hide actual content put there. What is the reason for this design decision?

jmstover commented 5 years ago

--containall doesn't do the normal sharing that we do. You still need a $HOME ... so, we create an empty $HOME ... we don't expect that you've already done so.... I mean, where you're running the container, the value of $HOME could change. Meaning, your $HOME could be /home/vsoch on one cluster, and /home/sochv on another.

If you have set something in the image for $HOME then use the --no-home option. That will not bind mount in your home as usual, or put an empty $HOME in there. But it will bind mount the current working directory. So, if your working directory was $HOME ... then $HOME will be brought in... but if it was: $HOME/projects/ABC123 ... then it's going to only bring in that directory.

EvanTheB commented 5 years ago

@jmstover thank you for the great explanation. It is frustrating to me that I am not able to work this out from the docs, and there are enough options interacting that I am not able to test every combination. Is there a plan to explain this somewhere, or some documentation that I have missed? All this also extends to the /tmp and scratch? options I think too.

I did not a know a $HOME location was required, but given it is, maybe somewhere less likely to interfere would be nice? Part of the issue is that trying to use -C and --bind ing something into $HOME will just silently not do anything. What is the difference between --no-home and the 'no home'-ness of -C?

For example in my first 5 seconds of singularity use I had an issue with .bashrc being sourced, so I turned on -C (which I had assumed would be the default behaviour), then I had to mount in ~/.ssh, but it just silently did nothing. Singularity 2 appears to at least warn about this (although my personal philosophy would be that if what the user asks for is impossible, error and exit, especially when the consequences are to do with files!)

jmstover commented 5 years ago

What is the difference between --no-home and the 'no home'-ness of -C?

What -C does with home is to not mount the host directory, but create an empty tmpfs location for $HOME within the container. This small tmpfs $HOME location is writable, where a normal image is not. Again, this fits within the "contain" part, in that we are not bringing anything from the Host into the container that we normally do by default.

What --no-home does, is it just skips the auto mounting of $HOME. We're just saying: "Don't put anything at the $HOME location."

What we are not doing here though is skipping the current working directory auto mount. So, whatever you current directory is, will be mounted into the container. Which could be your $HOME if that's your current location. It's just the explicit $HOME mount is skipped.

For example, if you had a small /home/user structure setup, you should be able to do something like (assume HOME=/home/user):

cd ~/projects/ABC123
singularity shell --no-home ./myimage.sif

And you would be in the image, with the ~/projects/ABC123 directory from the Host mounted, but still have whatever was in /home/user inside the image available to you.

EvanTheB commented 5 years ago

Thanks jmstover, I phrased my question badly;

Why is there a difference between --no-home and the 'no home'-ness of -C? My mental model of -C would be: "-C is the same as specifying -eip --no-home --no-tmp --minimal-dev..." But it seems like it is, "do things in a slightly different way than if you specified them separately", and "behave slightly differently if the user's pwd is in their $HOME".

Why is it impossible to --bind files into $HOME if -C is used? (or, if there is a reason, why is there no warning, and why is it not an error?)

vsoch commented 5 years ago

Some more comment from the peanut gallery :) I've now seen this kind of issue (not being clear about $HOME during build / after) pop up with users being confused in slack, on the list, and in GitHub issues. This is evidence that the information provided above isn't clearly stated in documentation. @jmstover the chance of the next person running into this thread and finding your response is close to nil - the action item here needs to be to make the documentation better. I still think it's a bug that I can put files in some home folder and then they just get erased (and no warning to this) but since this isn't my issue, it's not up to me to open a new one on the board. @EvanTheB if you think this behavior should be changed, you should do this. Thanks both for the discussion - I hope we see some action taken to resolve the confusion around what has been discussed.

jmstover commented 5 years ago

What --no-home does is skip the mounting of $HOME ... so nothing is created (unless you're in a working directory of $HOME). -C actually creates an empty $HOME, and mounts a tmpfs there for use as your $HOME.

The reason we can't use --bind to $HOME when -C is used, is $HOME is a tmpfs mount here. It gets mounted... But then we now want to bind something in after this empty tmpfs is mounted. And we don't have mount point in the image to do so. From debug output:

DEBUG   [U=1000,P=28419]   mountGeneric()                Mounting /home/jason/.ssh to /usr/local/singularity/3.2.0-rc2/var/singularity/mnt/session/final/home/jason/.ssh
VERBOSE [U=1000,P=28419]   mount()                       can't mount /home/jason/.ssh: no such file or directory

This is because $HOME is a mount in this case, and we only create the mount points that don't exist in the image structure... not mounts we're bringing in.

I've created an issue in the userdocs repo. It's at sylabs/singularity-userdocs#165 to try and get this started. @EvanTheB , if you can pop over to that documentation issue and expand on it if at all needed.

@cclerget and/or @bauerm97 can speak up with greater detail on this if I've messed up the explanation at all.

vsoch commented 5 years ago

Great, thanks @jmstover. Happy Friday! Hoping that all your tests are eating their greens, and ready for a good weekend break :)

EvanTheB commented 5 years ago

Thanks for the technical explanation of the behaviour @jmstover. I am trying to ask about DESIGN and lack of documentation, not the fact that there is a bug.

I will try to be more clear with my question.

Why is it not an error when a users requested bind mount cannot be honoured?

evaben@login-fpga ~$ singularity exec -C --bind $HOME/.ssh docker://debian bash -c "ls -a '$HOME'"
.  ..
evaben@login-fpga ~$ singularity --version
singularity version 3.0.2-1.el7.centos

Why was the warning in singularity 2.4 for the above behaviour removed?

singularity exec -C --bind $HOME docker://debian echo
WARNING: Not mounting requested bind point (already mounted in container): /home/evaben

evaben@evaben ~> singularity --version
2.4.2-dist
jmstover commented 5 years ago

@bauerm97 ^^^ Do you know the background on that change?