search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

import as non-root no longer possible #320

Closed flx42 closed 7 years ago

flx42 commented 7 years ago

I don't know if you are already aware of this.

The following used to work as non-root and setuid=no

mkdir ubuntu
singularity import ubuntu docker://ubuntu

It no longer works:

$ singularity -vvvv import ubuntu docker://ubuntu
increasing verbosity level (5)
ending argument loop
Exec'ing: /usr/local/libexec/singularity/cli/import.exec ubuntuVERBOSE [U=1000,P=1728]    message.c:52:message_init()                : Set messagelevel to: 5
DEBUG   [U=1000,P=1728]    privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=1000,P=1728]    privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
DEBUG   [U=1000,P=1728]    simage.c:34:main()                         : Running /usr/local/libexec/singularity/simage mount workflow
DEBUG   [U=1000,P=1728]    privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=1000,P=1728]    privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
VERBOSE [U=1000,P=1728]    mount.c:53:singularity_image_mount()       : Obtaining container name from environment variable
DEBUG   [U=1000,P=1728]    util/util.c:104:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_IMAGE'
VERBOSE [U=1000,P=1728]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_IMAGE'
DEBUG   [U=1000,P=1728]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_IMAGE
DEBUG   [U=1000,P=1728]    util/util.c:60:envar()                     : Checking environment variable length (<= 4096): SINGULARITY_IMAGE
DEBUG   [U=1000,P=1728]    util/util.c:66:envar()                     : Checking environment variable has allowed characters: SINGULARITY_IMAGE
VERBOSE [U=1000,P=1728]    util/util.c:89:envar()                     : Obtained input from environment 'SINGULARITY_IMAGE' = 'ubuntu'
DEBUG   [U=1000,P=1728]    privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=1000,P=1728]    privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
VERBOSE [U=1000,P=1728]    config_parser.c:54:singularity_config_open(): Opening configuration file: /usr/local/etc/singularity/singularity.conf
DEBUG   [U=1000,P=1728]    sessiondir.c:60:singularity_sessiondir_init(): Checking Singularity configuration for 'sessiondir prefix'
DEBUG   [U=1000,P=1728]    config_parser.c:83:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=1000,P=1728]    util/util.c:104:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_SESSIONDIR'
VERBOSE [U=1000,P=1728]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_SESSIONDIR'
DEBUG   [U=1000,P=1728]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_SESSIONDIR
VERBOSE [U=1000,P=1728]    util/util.c:56:envar()                     : Environment variable is NULL: SINGULARITY_SESSIONDIR
DEBUG   [U=1000,P=1728]    config_parser.c:111:singularity_config_get_value(): Called singularity_config_get_value(sessiondir prefix)
DEBUG   [U=1000,P=1728]    config_parser.c:127:singularity_config_get_value(): No configuration file entry found for 'sessiondir prefix'
DEBUG   [U=1000,P=1728]    sessiondir.c:75:singularity_sessiondir_init(): Set sessiondir to: /tmp/.singularity-session-1000.2065.28314269
DEBUG   [U=1000,P=1728]    util/file.c:245:s_mkpath()                 : Creating directory: /tmp/.singularity-session-1000.2065.28314269
DEBUG   [U=1000,P=1728]    sessiondir.c:91:singularity_sessiondir_init(): Opening sessiondir file descriptor
DEBUG   [U=1000,P=1728]    sessiondir.c:97:singularity_sessiondir_init(): Setting shared flock() on session directory
DEBUG   [U=1000,P=1728]    util/util.c:94:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOSESSIONCLEANUP
VERBOSE [U=1000,P=1728]    util/util.c:96:envar_defined()             : Environment variable is undefined: SINGULARITY_NOSESSIONCLEANUP
DEBUG   [U=1000,P=1728]    util/util.c:94:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOCLEANUP
VERBOSE [U=1000,P=1728]    util/util.c:96:envar_defined()             : Environment variable is undefined: SINGULARITY_NOCLEANUP
VERBOSE [U=1000,P=1728]    fork.c:74:singularity_fork()               : Forking child process
VERBOSE [U=1000,P=1728]    fork.c:90:singularity_fork()               : Hello from parent process
DEBUG   [U=1000,P=1728]    fork.c:109:singularity_fork()              : Assigning sigaction()s
DEBUG   [U=1000,P=1728]    fork.c:140:singularity_fork()              : Creating generic signal pipes
DEBUG   [U=1000,P=1728]    fork.c:148:singularity_fork()              : Creating sigcld signal pipes
DEBUG   [U=1000,P=1728]    fork.c:170:singularity_fork()              : Waiting on signal from watchdog
VERBOSE [U=1000,P=1730]    fork.c:78:singularity_fork()               : Hello from child process
DEBUG   [U=1000,P=1730]    fork.c:81:singularity_fork()               : Closing watchdog write pipe
DEBUG   [U=1000,P=1730]    fork.c:86:singularity_fork()               : Child process is returning control to process thread
DEBUG   [U=1000,P=1730]    config_parser.c:83:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=1000,P=1730]    config_parser.c:143:singularity_config_get_bool(): Called singularity_config_get_bool(allow user ns, 1)
DEBUG   [U=1000,P=1730]    config_parser.c:111:singularity_config_get_value(): Called singularity_config_get_value(allow user ns)
DEBUG   [U=1000,P=1730]    config_parser.c:127:singularity_config_get_value(): No configuration file entry found for 'allow user ns'
DEBUG   [U=1000,P=1730]    config_parser.c:162:singularity_config_get_bool(): Undefined configuration for 'allow user ns', returning default: yes
DEBUG   [U=1000,P=1730]    user.c:77:singularity_ns_user_unshare()    : Attempting to virtualize the USER namespace
DEBUG   [U=65534,P=1730]   user.c:92:singularity_ns_user_unshare()    : Enabled user namespaces
DEBUG   [U=65534,P=1730]   user.c:95:singularity_ns_user_unshare()    : Setting setgroups to: 'deny'
DEBUG   [U=65534,P=1730]   user.c:100:singularity_ns_user_unshare()   : Updating setgroups: /proc/1730/setgroups
DEBUG   [U=65534,P=1730]   user.c:113:singularity_ns_user_unshare()   : Setting GID map to: '0 1000 1'
DEBUG   [U=65534,P=1730]   user.c:118:singularity_ns_user_unshare()   : Updating the parent gid_map: /proc/1730/gid_map
DEBUG   [U=65534,P=1730]   user.c:131:singularity_ns_user_unshare()   : Setting UID map to: '0 1000 1'
DEBUG   [U=65534,P=1730]   user.c:136:singularity_ns_user_unshare()   : Updating the parent uid_map: /proc/1730/uid_map
DEBUG   [U=1000,P=1730]    config_parser.c:83:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=1000,P=1730]    config_parser.c:143:singularity_config_get_bool(): Called singularity_config_get_bool(mount slave, 1)
DEBUG   [U=1000,P=1730]    config_parser.c:111:singularity_config_get_value(): Called singularity_config_get_value(mount slave)
VERBOSE [U=1000,P=1730]    config_parser.c:119:singularity_config_get_value(): Got config key mount slave (= 'yes')
DEBUG   [U=1000,P=1730]    config_parser.c:149:singularity_config_get_bool(): Return singularity_config_get_bool(mount slave, 1) = 1
DEBUG   [U=1000,P=1730]    privilege.c:142:singularity_priv_escalate(): Not escalating privileges, user namespace enabled
DEBUG   [U=1000,P=1730]    mnt.c:54:singularity_ns_mnt_unshare()      : Virtualizing FS namespace
DEBUG   [U=1000,P=1730]    mnt.c:61:singularity_ns_mnt_unshare()      : Virtualizing mount namespace
DEBUG   [U=1000,P=1730]    mnt.c:70:singularity_ns_mnt_unshare()      : Making mounts slave
DEBUG   [U=1000,P=1730]    privilege.c:169:singularity_priv_drop()    : Not dropping privileges, user namespace enabled
DEBUG   [U=1000,P=1730]    rootfs.c:71:singularity_rootfs_init()      : Checking on container source type
DEBUG   [U=1000,P=1730]    config_parser.c:83:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=1000,P=1730]    rootfs.c:80:singularity_rootfs_init()      : Figuring out where to mount Singularity container
DEBUG   [U=1000,P=1730]    config_parser.c:111:singularity_config_get_value(): Called singularity_config_get_value(container dir)
VERBOSE [U=1000,P=1730]    config_parser.c:119:singularity_config_get_value(): Got config key container dir (= '/var/singularity/mnt')
VERBOSE [U=1000,P=1730]    rootfs.c:86:singularity_rootfs_init()      : Set image mount path to: /var/singularity/mnt
DEBUG   [U=1000,P=1730]    dir.c:44:rootfs_dir_init()                 : Inializing container rootfs dir subsystem
DEBUG   [U=1000,P=1730]    util/util.c:94:envar_defined()             : Checking if environment variable is defined: SINGULARITY_WRITABLE
VERBOSE [U=1000,P=1730]    util/util.c:99:envar_defined()             : Environment variable is defined: SINGULARITY_WRITABLE
DEBUG   [U=1000,P=1730]    rootfs.c:116:singularity_rootfs_mount()    : Checking 'container dir' mount location: /var/singularity/mnt
DEBUG   [U=1000,P=1730]    rootfs.c:127:singularity_rootfs_mount()    : Checking for rootfs_source directory: /var/singularity/mnt/source
DEBUG   [U=1000,P=1730]    rootfs.c:138:singularity_rootfs_mount()    : Checking for overlay_mount directory: /var/singularity/mnt/overlay
DEBUG   [U=1000,P=1730]    rootfs.c:149:singularity_rootfs_mount()    : Checking for overlay_final directory: /var/singularity/mnt/final
DEBUG   [U=1000,P=1730]    privilege.c:142:singularity_priv_escalate(): Not escalating privileges, user namespace enabled
DEBUG   [U=1000,P=1730]    dir.c:80:rootfs_dir_mount()                : Mounting container directory ubuntu->/var/singularity/mnt/source
DEBUG   [U=1000,P=1730]    privilege.c:169:singularity_priv_drop()    : Not dropping privileges, user namespace enabled
DEBUG   [U=1000,P=1730]    rootfs.c:181:singularity_rootfs_mount()    : OverlayFS enabled by host build
DEBUG   [U=1000,P=1730]    config_parser.c:83:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=1000,P=1730]    config_parser.c:143:singularity_config_get_bool(): Called singularity_config_get_bool(enable overlay, 1)
DEBUG   [U=1000,P=1730]    config_parser.c:111:singularity_config_get_value(): Called singularity_config_get_value(enable overlay)
VERBOSE [U=1000,P=1730]    config_parser.c:119:singularity_config_get_value(): Got config key enable overlay (= 'no')
DEBUG   [U=1000,P=1730]    config_parser.c:154:singularity_config_get_bool(): Return singularity_config_get_bool(enable overlay, 1) = 0
VERBOSE [U=1000,P=1730]    rootfs.c:184:singularity_rootfs_mount()    : Not enabling overlayFS via configuration
DEBUG   [U=1000,P=1730]    privilege.c:142:singularity_priv_escalate(): Not escalating privileges, user namespace enabled
VERBOSE [U=1000,P=1730]    rootfs.c:225:singularity_rootfs_mount()    : Binding the ROOTFS_SOURCE to OVERLAY_FINAL (/var/singularity/mnt/source->/var/singularity/mnt/final)
DEBUG   [U=1000,P=1730]    privilege.c:169:singularity_priv_drop()    : Not dropping privileges, user namespace enabled
DEBUG   [U=1000,P=1730]    rootfs.c:64:singularity_rootfs_dir()       : Returning singularity_rootfs_dir: /var/singularity/mnt/final
VERBOSE [U=1000,P=1730]    mount.c:70:singularity_image_mount()       : Setting SINGULARITY_ROOTFS to '/var/singularity/mnt/final'
DEBUG   [U=1000,P=1730]    rootfs.c:64:singularity_rootfs_dir()       : Returning singularity_rootfs_dir: /var/singularity/mnt/final
DEBUG   [U=1000,P=1730]    privilege.c:216:singularity_priv_drop_perm(): Called singularity_priv_drop_perm(void)
VERBOSE [U=1000,P=1730]    privilege.c:224:singularity_priv_drop_perm(): User namespace called, no privilges to drop
DEBUG   [U=1000,P=1730]    simage.c:34:main()                         : Running mount /usr/local/libexec/singularity/helpers/image-import.sh workflow
DEBUG   [U=1000,P=1730]    privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=1000,P=1730]    privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
DEBUG   [U=1000,P=1730]    privilege.c:216:singularity_priv_drop_perm(): Called singularity_priv_drop_perm(void)
DEBUG   [U=1000,P=1730]    privilege.c:233:singularity_priv_drop_perm(): Escalating permissison so we can properly drop permission
DEBUG   [U=1000,P=1730]    privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=1000)
ERROR   [U=1000,P=1730]    privilege.c:155:singularity_priv_escalate(): The feature you are requesting requires privilege you do not have
ABORT   [U=1000,P=1730]    privilege.c:156:singularity_priv_escalate(): Retval = 255
DEBUG   [U=1000,P=1728]    fork.c:52:handle_sigchld()                 : Checking child pids: 1730 1730
DEBUG   [U=1000,P=1728]    fork.c:54:handle_sigchld()                 : Forwarding signal through sigchld_signal_wpipe
DEBUG   [U=1000,P=1728]    fork.c:196:singularity_fork()              : Parent process is exiting
DEBUG   [U=1000,P=1728]    util/util.c:104:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_RUNDIR'
VERBOSE [U=1000,P=1728]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_RUNDIR'
DEBUG   [U=1000,P=1728]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_RUNDIR
VERBOSE [U=1000,P=1728]    util/util.c:56:envar()                     : Environment variable is NULL: SINGULARITY_RUNDIR
DEBUG   [U=1000,P=1728]    sessiondir.c:111:singularity_sessiondir_init(): Cleanup thread waiting on child...
DEBUG   [U=1000,P=1728]    sessiondir.c:116:singularity_sessiondir_init(): Checking to see if we are the last process running in this sessiondir
VERBOSE [U=1000,P=1728]    sessiondir.c:118:singularity_sessiondir_init(): Cleaning sessiondir: /tmp/.singularity-session-1000.2065.28314269
DEBUG   [U=1000,P=1728]    util/file.c:267:s_rmdir()                  : Removing directory: /tmp/.singularity-session-1000.2065.28314269

git bisect:

25ff39547468d49a64cc2d771099d6e8f3ad7542 is the first bad commit
commit 25ff39547468d49a64cc2d771099d6e8f3ad7542
Author: Michael Bauer <bauerm@umich.edu>
Date:   Mon Nov 7 12:41:14 2016 +0100

    Updated cli/*.exec to properly pass args to simage binary

:100644 100644 8d131cfc208c085f50a12aa044e996ca8d10eb50 efb28297eb416ae4f41bb76a2ea98b82af90b662 M      configure.ac
:040000 040000 100999e5998f2440b2bbb566655444c396012ba8 29858ed4b40a9242206d75339450226c93854be9 M      libexec
bauerm97 commented 7 years ago

Hi @flx42 , thanks for the report. We are aware that this is currently the case, and we are currently investigating the proper way to handle importing as non-root. We should have the same or a greater level of non-root import functionality back in the code some time soon, but we want to make sure that we eliminate all security concerns before introducing any more non-root functionality.

Also, is the debug output from master or from that specific git commit. If it's from that commit, please re run this from master and show me what you see. Thanks.

flx42 commented 7 years ago

With 1a83f9ceb8159155b089585765568ad4674afc80:

$ singularity -vvvv import ubuntu docker://ubuntu
increasing verbosity level (5)
ending argument loop
Exec'ing: /usr/local/libexec/singularity/cli/import.exec ubuntuVERBOSE [U=1000,P=13688]   message.c:52:message_init()                : Set messagelevel to: 5
DEBUG   [U=1000,P=13688]   privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=1000,P=13688]   privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
DEBUG   [U=1000,P=13688]   simage.c:55:main()                         : Running /usr/local/libexec/singularity/simage mount workflow
DEBUG   [U=1000,P=13688]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=1000)
ERROR   [U=1000,P=13688]   privilege.c:155:singularity_priv_escalate(): The feature you are requesting requires privilege you do not have
ABORT   [U=1000,P=13688]   privilege.c:156:singularity_priv_escalate(): Retval = 255
vsoch commented 7 years ago

Just curious, does it work when not using docker? If it does, could you try exporting SINGULARITY_CACHEDIR (or whatever argument goes into the command or config to specify cache) to something definitely owned by your user?

It's maybe not that, but just want to be extra sure since we added a bunch of new stuffs recently!

bauerm97 commented 7 years ago

@vsoch The reason it doesn't work right now is because in simage.c we call singularity_priv_escalate() before anything else. Since simage binary doesn't have SUID, this has the effect of checking for root. When you call singularity_priv_escalate() without SUID or when running as a regular user, it will abort due to it being unable to escalate. We could theoretically remove/move these lines in #322 , to enable certain workflows to attempt to run without root.

vsoch commented 7 years ago

Ah ok cool! Thanks for the details, I'll keep watching!

gmkurtzer commented 7 years ago

@bauerm97 is right about why it is not working now. I am considering removing that bit from simage.c because the idea of bootstrap'ing with SUID just opens us up to too many potential security risks (users manipulating the shell or python environment to run arbitrary code as root).

vsoch commented 7 years ago

+1

gmkurtzer commented 7 years ago

Import is no longer allowed for directories, but this feature works for images in the development branch. Closing issue. Comment if it needs to be reopened. Thanks!

flx42 commented 7 years ago

It doesn't seem to work on 5f5e3bc92f3d0aba07c060089585d4bb27da4cef (development branch) with the following: 

$ ./configure --disable-suid && make && sudo make install
$ singularity create -F trusty.img
Initializing Singularity image subsystem
Opening image file: trusty.img
Creating 768MiB image
Binding image to loop
ERROR  : The feature you are requesting requires privilege you do not have
ABORT  : Retval = 255

Without --disable-suid, it works, but probably because this path is broken (see #569):

$ ./configure && make  && sudo make install
$ singularity create -F trusty.img
Initializing Singularity image subsystem
Opening image file: trusty.img
Creating 768MiB image
Binding image to loop
Creating file system within image
Image is done: trusty.img
$ SINGULARITY_NOSUID=1 singularity create -F trusty.img
Initializing Singularity image subsystem
Opening image file: trusty.img
Creating 768MiB image
Binding image to loop
ERROR  : The feature you are requesting requires privilege you do not have
ABORT  : Retval = 255
gmkurtzer commented 7 years ago

To create a new container, it does require privileges, and it uses SUID for that which is why you are seeing this error. There is no workaround, aside from calling with sudo and allowing SUID.

Considering you need SUID in order to use Singularity images anyway, you may as well either use the SUID process flow or don't use images (just run a shell (or other action) on a docker URI):

$ singularity shell docker://centos
flx42 commented 7 years ago

Ok, so this isn't supported for images, I didn't understand that from your previous comment. fine for me.

flx42 commented 7 years ago

The bug in https://github.com/singularityware/singularity/issues/569 probably gave the illusion that it worked ;)

gmkurtzer commented 7 years ago

Yeah, exactly! The user namespace doesn't provide the means to deal with loop images, so if it worked, it was because there was some SUID code doing it's job.

Why not just use the SUID code? Honestly, the user namespace is flakey at best due to the fact that it is not ubiquitous or equally implemented across different distributions. Maybe one day it will become more standardized, but for now, all I hear from large production sites is that it is an unstable headache.

flx42 commented 7 years ago

On CentOS, don't even try, user namespaces are a preview feature and therefore largely unusable.

On other distros (like ubuntu 16.04), it works fine, and if you remap the UID you are root in the container, which allow you to do more things without being "real" root from the outside. From a security standpoint, we prefer to rely on user namespaces instead of SUID binaries, no offense ;)

I understand that it's not possible to create a Singularity image without privileges, and that doing a singularity import on a folder with SUID creates many security issues and that's why you removed this feature (right? or was there other reasons?). But in a full non-SUID setting (--disable-suid), folder import should work without any security concern.

gmkurtzer commented 7 years ago

Ubuntu and Debian's implementation of user namespaces are not even equal though! I stopped putting conditionals in, so Singularity mostly works with user namespace on Ubuntu (as long as you don't use images), but not always on Debian. It is quite frustrating... Luckily, almost no big HPC centers use Debian or Ubuntu. I don't mean to alienate any of the *.deb-heads out there, but I did a check on the top500, your DGX was the only one!

The problem is that the user namespaces, even as Ubuntu implemented it, isn't quite enough for all of the HPC features most centers rely on. While, I would also prefer to discard the security baggage of SUID, it just isn't possible in the production world. :(

Yes, you are 100% correct about doing the import on a folder with SUID, and thus it is disabled.

You are also right about import "should" work without any security concern, but... import is an image specific function. Plus, I have an OCD about writing files to the file system that have incorrect permissions (and if splatting them out as a user, every file is wrong).

flx42 commented 7 years ago

Plus, I have an OCD about writing files to the file system that have incorrect permissions (and if splatting them out as a user, every file is wrong).

That's what docker shell docker://image is already doing :), but by caching layers to $HOME/.singularity, and extracting them to /tmp/.singularity-runtime.XXXXXXXX/. This path also works for us, but it's slower since you pay the extraction cost at every container start (and we have large images). Now the only option is to docker export to a folder, then singularity shell ~/myrootfs/

gmkurtzer commented 7 years ago

Yep, but by definition it is not persistent. It is the persistency of wrong permissions that makes it intolerable to me! But ... with that said, @vsoch and I have been discussing a pull sub command that might do exactly what you are thinking. :)