No engine config chunk provided - Singularity error

[rcorces]

Our IT department recently upgraded our Singularity version to patch some security issues. In doing so, it seems like something has broken. This is happening for the PEPATAC crate and I debated posting this there but if my intuition is correct then it may be a bulker issue. Any time a job is attempted to be launched or a command is run inside the bulker crate, I get this error coming from Singularity: ERROR : No engine config chunk provided

Relatively recently, Singularity changed how config files are handled and introduced the code corresponding to this error message: https://github.com/sylabs/singularity/commit/547399010df92ebdc8387612886cf79f2363673c

It seems like Singularity expects a specific env variable and at least on my system, that isnt set. This could be a user-specific problem with how IT installed the new Singularity, or it could be some problem with how Bulker is communicating with Singularity.

For what its worth, Singularity does work just fine with:

singularity exec library://alpine cat /etc/alpine-release

I'm now using singularity version 3.8.0-1.el8 no idea what my previous version was but certainly earlier than the above-referenced commit.

Have you guys tested Bulker on more recent versions of Singularity?

[nsheff]

We're using it on Singularity 3.7.1 -- I have used it on several of the early versions (I think maybe 3.4, 3.6). I haven't tried with 3.8.

If you do:

cat `which cat`

what do you see? Can you try executing that code directly in the shell, to see if you can run it if it's not wrapped in a script?

[rcorces]

databio/pepatac|~$ which cat
/corces/home/shared/tools/bulker/bulker_crates/databio/pepatac/1.0.4/cat
databio/pepatac|~$ cat `which cat`
ERROR  : No engine config chunk provided

[nsheff]

Ok sorry, you're going to have to use the native cat rather than the bulker cat...

/usr/bin/cat `which cat`

Or from outside bulker:

cat /corces/home/shared/tools/bulker/bulker_crates/databio/pepatac/1.0.4/cat

[rcorces]

databio/pepatac|~$ /usr/bin/cat `which cat`
#!/bin/sh

if [ ! -f "/corces/home/shared/tools/singularity/simages/databio/alpine-coreutils" ]; then
  singularity pull alpine-coreutils docker://databio/alpine-coreutils
  mv alpine-coreutils /corces/home/shared/tools/singularity/simages/databio/alpine-coreutils
fi

LC_ALL=C singularity exec \
  -B "/corces:/corces" \
  -B "/dev:/dev" \
  -B "/tmp:/tmp" \
  -B "/opt:/opt" \
  /corces/home/shared/tools/singularity/simages/databio/alpine-coreutils cat "$@"

[nsheff]

Ok.

Can you try executing that code directly in the shell, to see if you can run it if it's not wrapped in a script?

What I mean by that is, try running something like this:

LC_ALL=C singularity exec \
  -B "/corces:/corces" \
  -B "/dev:/dev" \
  -B "/tmp:/tmp" \
  -B "/opt:/opt" \
  /corces/home/shared/tools/singularity/simages/databio/alpine-coreutils cat /etc/alpine-release

[rcorces]

that seems to work?

(p3.8.5) [rcorces@pelayo ~]$ LC_ALL=C singularity exec \
>   -B "/corces:/corces" \
>   -B "/dev:/dev" \
>   -B "/tmp:/tmp" \
>   -B "/opt:/opt" \
>   /corces/home/shared/tools/singularity/simages/databio/alpine-coreutils cat /etc/alpine-release
3.11.3

[nsheff]

Ok... so you're saying that chunk of code can be executed directly, but if you put it in a script and try to run the script, it doesn't work... right? Just to double-check, paste that code into a script cat.sh, set permissions, and try running it: ./cat.sh -- can you do that? It should break, I guess.

Try replacing #!/bin/sh with your path to bash. maybe you have those required env vars you mentioned set in bash only?

If that's the case, this can be fixed by adjusting those bulker templates to use bash in your shebang line instead of sh.

[rcorces]

Perhaps I mis-interpreted your previous request. When I run the LC_ALL... command within the bulker crate, it fails with the same error:

databio/pepatac|~$ LC_ALL=C singularity exec \
>   -B "/corces:/corces" \
>   -B "/dev:/dev" \
>   -B "/tmp:/tmp" \
>   -B "/opt:/opt" \
>   /corces/home/shared/tools/singularity/simages/databio/alpine-coreutils cat /etc/alpine-release
ERROR  : No engine config chunk provided

This error happens within the bulker crate regardless of whether it is put in a script or executed from command line.

Outside of the crate, both script and command line work just fine.

[rcorces]

I'm no longer convinced this is a singularity version problem. It seems to work fine on another server with an updated version of singularity. I'll keep digging.

[nsheff]

Interesting.

Yes I don't think it's singularity version... but has something to do with the environment. When you activate the crate, the only thing that happens is that your PATH variable changes, really. And, the prompt changes. I think that's basically it. So, something about the PATH variable changing is causing that. You could inspect the PATH variable before and after starting the crate to see what the changes are, and try doing those changes without activating a crate, to see if just changing the PATH is enough to reproduce it.

[nsheff]

is the crate activation somehow changing the accessibility of a global singularity config file?

[rcorces]

like any good end user, I messed with a whole bunch of stuff all at once and now it works and I have no idea why. For what its worth, just prepending /corces/home/shared/tools/bulker/bulker_crates/databio/pepatac/1.0.4 to my path would result in the weird "ERROR : No engine config chunk provided". I reinstalled bulker, deleted all of my crates, reinstalled the pepatac crate using the latest version, probably did a few other things in the process, and now it works.

sorry for the hassle.