SIGTTOU sent to parent perl (system) or python (subprocess) on second call to `singularity exec` within a script

[wresch]

Version of Singularity:

3.0.1-1

Expected behavior

Doing a singularity exec from perl/python twice should work

Actual behavior

A single singularity exec from perl (system) or python (subprocess.call) runs as expected. A second call results in the calling script getting sent a SIGTTOU and being stopped. 2.6.0 did not exhibit this behavior

Steps to reproduce behavior

Get a container for testing

singularity pull docker://godlovedc/lolcow

Perl script:

my @cmd = ("singularity", "exec", "lolcow_latest.sif", "ls", "--version");
system(@cmd); 
system(@cmd);

Python script:

import subprocess

cmd = ["singularity", "exec", "lolcow_latest.sif", "ls", "--version"]
subprocess.call(cmd)
subprocess.call(cmd)

Running either one of those:

$ python test.py
ls (GNU coreutils) 8.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard M. Stallman and David MacKenzie.

[1]+  Stopped                 python foo.py

$ ps -u $USER -o 'pid,ppid,state,command'
  PID  PPID S COMMAND
11138  8854 T python foo.py
$ kill -9 11138

Same result with perl. If you comment out the second call to singularity the script finishes as expected. If you call a different container, the script gets stopped as it did above. Here is an excerpts of an strace looking only at the python process:

[...snip...]
munmap(0x2aaab238d000, 4096)            = 0
close(3)                                = 0
wait4(11516, ls (GNU coreutils) 8.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard M. Stallman and David MacKenzie.
[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11516
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11516, si_uid=913, si_status=0, si_utime=3, si_stime=3} ---
pipe([3, 4])                            = 0
fcntl(3, F_GETFD)                       = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fcntl(4, F_GETFD)                       = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2aaaaacb3050) = 11550
close(4)                                = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaab238d000
read(3, "", 1048576)                    = 0
mremap(0x2aaab238d000, 1052672, 4096, MREMAP_MAYMOVE) = 0x2aaab238d000
munmap(0x2aaab238d000, 4096)            = 0
close(3)                                = 0
wait4(11550, 0x7fffffffbb04, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_STOPPED, si_pid=11550, si_uid=913, si_status=SIGTTOU, si_utime=1, si_stime=0} ---
--- SIGTTOU {si_signo=SIGTTOU, si_code=SI_KERNEL} ---
--- stopped by SIGTTOU ---

And indeed if we ignore SIGTTOU the python script finishes just fine

import subprocess
import signal

signal.signal(signal.SIGTTOU, signal.SIG_IGN)

cmd = ["singularity", "exec", "lolcow_latest.sif", "ls", "--version"]
subprocess.call(cmd)
subprocess.call(cmd)

My (admittedly weak) understanding of SIGTTOU is that it is sent to background processes that tries to modify terminal settings or write to stderr/stdout iff the tty has tostop mode enabled. But the python/perl processes here are not in fact background processes and my tty (like most, I belive) does not have tostop mode enabled.

[jscook2345]

I was able to reproduce both the error and the work around. @cclerget

[cclerget]

@jscook2345 Working on it

[cclerget]

@wresch #2366 should fix the issue. If you can give it a try, thanks !

[wresch]

works for me:

$ singularity --version
singularity version 3.0.1-55.gc52352b

$ python test.py
ls (GNU coreutils) 8.25
...

ls (GNU coreutils) 8.25
...

$