shell /bin/sh does not exist in container when running with singularity

[westonkjones]

Hello! I'm trying to run the pipeline with the singularity flag, but I get the following error (anonymized for brevity)

[myuser@login001 ~]$ caper run /home/myuser/encodedcc/rna-seq-pipeline:1.2.4/rna-seq-pipeline.wdl -i /data/user/myuser/experiments/myexperiment/samples/E1/output/2021-10-10_19_53_26/input.json --metadata-output singu.metadata --cromwell-stdout singu.stdout --max-retries 0 --singularity
2021-10-10 23:07:55,744|caper.caper_base|INFO| Creating a timestamped temporary directory. /data/user/home/myuser/.caper_tmp/rna-seq-pipeline/20211010_230755_743396
2021-10-10 23:07:55,744|caper.caper_runner|INFO| Localizing files on work_dir. /data/user/home/myuser/.caper_tmp/rna-seq-pipeline/20211010_230755_743396
2021-10-10 23:08:12,442|caper.caper_workflow_opts|INFO| Singularity image found in WDL's metadata. wdl=/home/myuser/encodedcc/rna-seq-pipeline:1.2.4/rna-seq-pipeline.wdl, s=encodedcc/rna-seq-pipeline:1.2.4
2021-10-10 23:08:15,885|caper.singularity|INFO| Building local singularity image for encodedcc/rna-seq-pipeline:1.2.4
WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
FATAL:   shell /bin/sh doesn't exist in container
Traceback (most recent call last):
  File "/home/myuser/.local/bin/caper", line 13, in <module>
    main()
  File "/home/myuser/.local/lib/python3.8/site-packages/caper/cli.py", line 675, in main
    return runner(parsed_args, nonblocking_server=nonblocking_server)
  File "/home/myuser/.local/lib/python3.8/site-packages/caper/cli.py", line 226, in runner
    subcmd_run(c, args)
  File "/home/myuser/.local/lib/python3.8/site-packages/caper/cli.py", line 347, in subcmd_run
    thread = caper_runner.run(
  File "/home/myuser/.local/lib/python3.8/site-packages/caper/caper_runner.py", line 410, in run
    options = self._caper_workflow_opts.create_file(
  File "/home/myuser/.local/lib/python3.8/site-packages/caper/caper_workflow_opts.py", line 234, in create_file
    s.build_local_image()
  File "/home/myuser/.local/lib/python3.8/site-packages/caper/singularity.py", line 56, in build_local_image
    return check_call(cmd, env=env)
  File "/share/apps/rc/software/Anaconda3/2020.11/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['singularity', 'exec', 'encodedcc/rna-seq-pipeline:1.2.4', 'echo', 'Built local singularity image for encodedcc/rna-seq-pipeline:1.2.4']' returned non-zero exit status 255.

When I search for that error there isn't much to go on, but people seem to mention the image was built for the wrong version. I've tried to module load few different Singularity versions on my SLURM cluster, but none have worked. This happens when trying to run on the test files, and when I tried with version 1.2.3

[ottojolanki]

Hi. I will look into it. Which singularity versions have you tried?

[leepc12]

Please run the following for debugging and post outputs.

$ singularity exec encodedcc/rna-seq-pipeline:1.2.4 echo hello

Also, please try with the latest caper>=2.0.0. That will solve most singularity/conda/slurm problems

$ pip install caper --upgrade

[ottojolanki]

Thanks for pitching in @leepc12. @westonkjones let us know how it goes and we'll help you if this does not already resolve your issue. Thanks!