Closed kelly-sovacool closed 1 week ago
able to reproduce this with:
singularity shell -B /lscratch,/data/CCBR_Pipeliner,/data/sovacoolkl,/gpfs/gsfs10/users/CCBR_Pipeliner,/gpfs/gsfs12/users/sovacoolkl docker://nci
ccbr/charlie_star_ucsc_cufflinks:v0.4.0 zsh
python
import pandas
pandas
one workaround:
python -E
import pandas
pandas
we may want to use singularity -C
or singularity -E
, but will that mess up $SLURM_JOB_ID
which is needed for temp dirs in some rules?
when using singularity -C
, $SLURM_JOB_ID
is not available in the containers. However, snakemake calls singularity exec
, which does not have the --env
argument for setting the $SLURM_JOB_ID
in the container. Also, it would likely use the value of $SLURM_JOB_ID
from the master job instead of from within the individual rule jobs.
trying python -E
inside the rule definition as a workaround.
possible solutions:
singularity -C
- can't use it, $SLURM_JOB_ID won't be availablepython -E
- works but only when we explicitly call python, may not work when using tools that have python dependencies$PYTHONPATH
inside container - WORKS. tested with docker://nciccbr/charlie_star_ ucsc_cufflinks:v0.4.1
. https://github.com/CCBR/CHARLIE/commit/1fc7dc01d7456ba53f59752cf75075e768372cf3$PYTHONPATH
at beginning of snakemake rule - almost definitely would work, but would be cumbersome and clutter up snakemake rules.