Tracing a Singularity container

[gkiar]

Hi,

I'm trying to use reprozip to monitor a workflow of mine, but getting a non-very-descriptive error message below when doing this. I'm able to run simpler commands, such as echo hello, but my current command (which happens to be a Python tool which launches Singularity containers) returns the following response each time. I'm not sure what the issue with the executable I'm currently trying to run is, or why reprozip seems incompatible with it, but am eager to see if you may know why this is happening.

Thanks for your help!

$ reprozip trace -w --dir=/mydir/ command arg1 arg2 arg3 arg4 ...
[REPROZIP] 15:14:44.630 CRITICAL: cleaning up, 4 processes to kill...
Traceback (most recent call last):
  File "/project/6008063/gkiar/computing/env/closh/lib/python3.6/site-packages/reprozip/main.py", line 390, in main
    status = args.func(args)
  File "/project/6008063/gkiar/computing/env/closh/lib/python3.6/site-packages/reprozip/main.py", line 206, in trace
    args.verbosity)
  File "/project/6008063/gkiar/computing/env/closh/lib/python3.6/site-packages/reprozip/tracer/trace.py", line 341, in trace
    c = _pytracer.execute(binary, argv, database.path, verbosity)
_pytracer.Error: Error occurred

[remram44]

Oh no!

Unfortunately not all of the C code produces meaningful error messages (no exceptions in C!), but hopefully the log should have more information? (look for ~/.reprozip/log or re-run as reprozip -v -v trace ...)

It is very probable that something bad happens when running something that uses namespaces itself such as Singularity, but perhaps something can be done.

Thanks for reporting this to us!

[gkiar]

Thanks for the quick response, @remram44 - attached is my log file from running with -v -v and piping stderr (i.e. 2> log.txt). Thanks!

[remram44]

I think I found it! 1c4adb39 should fix. Terribly sorry about this overlook.

Is there a way you can update from git? Otherwise I will do another release tomorrow.

[gkiar]

No problem, I'll update from git and test shortly! Thank you for your incredibly quick response! :)

[gkiar]

@remram44 progress!! I'm able to launch the task, now the issue I'm encountering just seems to be a user permissions issue when launching the Singularity container (Singularity Error below; tagging @vsoch who may know more). Would it be possible to run reprozip inheriting access of my user account? Thanks so much!

ERROR  : Singularity is not running with appropriate privileges!
ERROR  : Check installation path is not mounted with 'nosuid', and/or consult manual.
ABORT  : Retval = 255

[remram44]

suid will not work while the process is being traced. This is a safety feature in Linux (if the process can gain privileges via suid and ReproZip is still tracing it, it could subvert it to run whatever command as root)

Workaround is running the whole thing as root, or I could change ReproZip to stop tracing when it reaches a suid executable (but then none of those processes would be captured).

[gkiar]

I see, that makes complete sense! It would be fantastic if reprozip could trace containerized executions, so I wonder if @vsoch or others on that team would be able to loosen the suid restriction in some way... I'll cross-list this on one of their repos and see if there is anything we could do.

Thanks!

[remram44]

Are you running on a machine where using sudo is not an option?

[gkiar]

Correct - I'm running on Compute Canada (a public shared resource), and trying to develop a tool that will work regardless of a user's admin privileges. The goals is (more or less) to launch and record all sorts of provenance information from container-encapsulated pipelines, including the file I/O mapping performed by reprozip

[remram44]

I'm wondering if having ReproZip not trace Singularity would help you in any way?

Reprozip would detach, which would allow Singularity to run, and you would still capture whatever script called Singularity. However Singularity itself and everything happening in the container would not be captured.

[gkiar]

Hi @remram44 in particular I'd like to capture what is happening in the container most of all, as that's where the real "meat" of processing occurs. I don't care so much about the environment loading in particular, but certainly the processing within. Thanks for asking!

[remram44]

I have fixed the original bug and added a warning (#296) when a set-uid executable is encountered, so that users know why the program is failing.

Unfortunately I don't see what else the Singularity team or us can do, since we cannot work around Linux security features. I am going to go ahead with the release, please don't hesitate to let us know if you hit any other issue or roadblock in the future!

[gkiar]

Thank you very much!