Run command with Singularity will miss python packages.

[Autumn-Roy]

Describe the bug When run singularity run -B $(pwd)/sample_data/input_data:/data -B $(pwd)/reference:/reference -B $(pwd)/results:/results xenocp.sif /data/inputs.yml as README, it will occur error: ModuleNotFoundError: No module named 'cwltool'

To Reproduce

1. git clone https://github.com/stjude/XenoCP.git
2. cd XenoCP
3. docker build -t xenocp .
4. SINGULARITY_NOHTTPS=1 singularity build my_username_app_img.sif docker-daemon://xenocp:latest
5. wget -r -np -R "index.html*" -nH --cut-dirs=3 http://ftp.stjude.org/pub/software/xenocp/reference/MGSCv37
6. singularity run -B $(pwd)/sample_data/input_data:/data -B $(pwd)/reference:/reference -B $(pwd)/results:/results xenocp.sif /data/inputs.yml

Expected behavior The command running successfully.

Screenshots

Environment (please complete the following information):

• OS: CentOS
• Version 7.7.1908
• Runner singularity

Additional context Add any other context about the problem here.

[adthrasher]

The CWL installation is under /root, which will not be readable when using singularity as the container runs as your user. We don't actually provide any instructions for running with singularity for XenoCP because of these dependencies.

That said, I've pushed an updated Dockerfile that should remove root dependencies. I was able to run through the example with singularity. Please pull the latest commits and try rebuilding the docker image and singularity image.

[Autumn-Roy]

@adthrasher Thanks, the latest commits can support singularity now!! I have another problem is connection with https://schema.org/version/latest/schemaorg-current-https.rdf will be failed. Because I have to use vpn to get this website. Can I configure the net agent in somewhere?

[adthrasher]

The cwl-runner has an option to disable loading of the schemas, which may work for your environment. You could try adding that to the ENTRYPOINT directive in the Dockerfile.

  --skip-schemas        Skip loading of schemas

[Autumn-Roy]

The singularity works, but it paused on the step :[workflow ] starting step mapping-bwa-mem for 4 hours when I run the test command in README.

The command is below:

export XENOCP_HOME=~/XenoCP
singularity run -B ${XENOCP_HOME}/sample_data/input_data:/data -B ${XENOCP_HOME}/reference:/reference -B ${XENOCP_HOME}/results:/results xenocp.sif /data/inputs.yml

[adthrasher]

What sort of machine are you running on? It does rely on multi-threading, so a machine with few processors and insufficient RAM will likely require significantly more run time. On the node that I tested, the singularity implementation completed the test command in 7 minutes.

[Autumn-Roy]

My system is Centos 7, CPU: 2 x Intel Xeon Scalable Cascade Lake 6248(2.5GHz, 20 cores), Mem:12 x Samsung 16GB DDR4 ECC REG 2666. Because we use HPC slurm controler, so I apply 4 cores for this job.

[Autumn-Roy]

I applied 32 cores with 256GB memory, and the example finished in 10 minutes. Thanks for your reply.