singularity shell segfault with docker://centos:6

[jbd]

Hi,

I'm running Singularity 2.3.1-development.ga78190c8 on a Debian testing using kernel 4.11.0-1-amd64.

$ singularity --version
2.3.1-development.ga78190c8
$ singularity selftest
 + sh -c test -f /usr/local/etc/singularity/singularity.conf                           (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/action-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/create-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/expand-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/export-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/import-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/mount-suid                               (retval=0) OK

I've got a segfault with a centos6 docker image:

$ singularity shell docker://centos:6 
Docker image path: index.docker.io/library/centos:6
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Segmentation fault

I've got no problem with a centos7 docker image:

$ singularity --version
2.3.1-development.ga78190c8
$ singularity shell docker://centos:7
Docker image path: index.docker.io/library/centos:7
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Singularity centos:7:~> 

Here is the tail of the debug output with the centos6 container:

[...]
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning NULL on 'TARGET_PWD'
DEBUG   [U=21489,P=13302]  main()                                    Attempting to chdir to CWD: /home/jbdenis
DEBUG   [U=21489,P=13302]  envar_set()                               Setting environment variable: 'SINGULARITY_CONTAINER' = 'centos:6'
DEBUG   [U=21489,P=13302]  envar_set()                               Setting environment variable: 'SINGULARITY_NAME' = 'centos:6'
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning NULL on 'SHELL'
DEBUG   [U=21489,P=13302]  envar_set()                               Unsetting environment variable: SINGULARITY_SHELL
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning value from registry: 'COMMAND' = 'shell'
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning value from registry: 'COMMAND' = 'shell'
LOG     [U=21489,P=13302]  main()                                    USER=jbdenis, IMAGE='centos:6', COMMAND='shell'
DEBUG   [U=21489,P=13302]  action_shell()                            Exec'ing /.singularity.d/actions/shell
VERBOSE [U=21489,P=24203]  main()                                    Cleaning directory: /tmp/.singularity-runtime.1U1fKtq0
DEBUG   [U=21489,P=24203]  s_rmdir()                                 Removing directory: /tmp/.singularity-runtime.1U1fKtq0
Segmentation fault (core dumped)

(full output https://gist.github.com/jbd/6b6983514e6c430d2c8db147af025f66)

[jbd]

Seems to be related to https://github.com/docker/for-linux/issues/58, see this comment in particular.

[jbd]

I confirm passing vsyscall=emulate to the kernel resolves it for me.

In /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="vsyscall=emulate"

# update-grub && reboot

I let you close the issue if you want to add something.

[gmkurtzer]

Ahh, great catch! We should update our docs. @vsoch, do you want to add a troubleshooting entry for this or shall I?

Thanks!

[vsoch]

@gmkurtzer I'll get us started here, and if you want to edit / adjust I can add to our troubleshooting docs. Here is a first go not following this issue closely:

Segfault on Bootstrap of Centos Image

If you are bootstrapping a centos 6 docker image, you might hit a segfault:

$ singularity shell docker://centos:6 
Docker image path: index.docker.io/library/centos:6
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Segmentation fault

@gmkurtzer add something here for why this is happening?

The fix is to pass the variable vsyscall=emulate to the kernel, meaning in the file /etc/default/grub, add the following

GRUB_CMDLINE_LINUX_DEFAULT="vsyscall=emulate"

and then update grub and reboot:

update-grub && reboot

@gmkurtzer and @jbd were these commands on the host, or somehow executed to the image? Thanks for the help!

[jbd]

These commands were on the host. The /etc/default/grub is debian specific, you might want to mention that.

It would be interesting to know the vsyscall status for the main GNU/Linux distributions.

On the top of my head, I would suggest:

$ grep VSYSCALL /boot/config*

or

$  zgrep VSYSCALL /proc/config.gz # if it exists of course

From the debian kernel changelog:

linux (4.10~rc6-1~exp1) experimental; urgency=medium
[amd64] Enable LEGACY_VSYSCALL_NONE instead of LEGACY_VSYSCALL_EMULATE
    (Closes: #852620).  This breaks (e)glibc versions < 2.14 and dietlibc
    versions < 0.33.  It can be reverted using the kernel parameter:
    vsyscall=emulate

[vsoch]

ok check this out, hope I got it right! http://singularity.lbl.gov/faq#segfault-on-bootstrap-of-centos-image

[jbd]

Seems OK to me, thank you for the update !

Also, from the kernel-parameters.txt of linux kernel source tree:

vsyscall=   [X86-64]
            Controls the behavior of vsyscalls (i.e. calls to
            fixed addresses of 0xffffffffff600x00 from legacy
            code).  Most statically-linked binaries and older
            versions of glibc use these calls.  Because these
            functions are at fixed addresses, they make nice
            targets for exploits that can control RIP.

            emulate     [default] Vsyscalls turn into traps and are
                        emulated reasonably safely.

            native      Vsyscalls are native syscall instructions.
                        This is a little bit faster than trapping
                        and makes a few dynamic recompilers work
                        better than they would in emulation mode.
                        It also makes exploits much easier to write.

            none        Vsyscalls don't work at all.  This makes
                        them quite hard to use for exploits but
                        might break your system.

The change to "emulate" has security implications. Given that emulate is the default mode of the upstream. I guess it is OK to change it. I think Singularity users should be clearly aware of that. Maybe it should appears somewhere in the Security section of the documentation or this linked issue is enough for the moment ? Your call ! =)

[vsoch]

okay, just added a note about it to the same troubleshooting page. Thanks!

[jbd]

I think this should not appear on the troubleshooting section:

@gmkurtzer I’ll get us started here, and if you want to edit / adjust I can add to our troubleshooting docs. Here is a first go not following this issue closely:

[vsoch]

I have no idea what you are talking about! :wink:

[dtrudg]

Closing, as present in docs at http://singularity.lbl.gov/faq#segfault-on-bootstrap-of-centos-image