search

CCBR / CHARLIE

Circrnas in Host And viRuses anaLysis pIpEline for Detection Annotation Quantification of circRNAs
https://ccbr.github.io/CHARLIE/
MIT License
2 stars 1 forks source link

clashing python libraries in singularity containers and ccbrpipeliner module #120

Closed kelly-sovacool closed 1 week ago

kelly-sovacool commented 1 week ago 
Activating singularity image /vf/users/guibletwm/charlietest/charlie/.snakemake/singularity/85508a5ad215e06ac7e60750d9a291aa.simg
+ cd /vf/users/guibletwm/charlietest/charlie/ref
+ samtools faidx /vf/users/guibletwm/charlietest/charlie/ref/ref.fa
+ cut -f1-2 /vf/users/guibletwm/charlietest/charlie/ref/ref.fa.fai
+ python /gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/CHARLIE/.v0.11.0/workflow/scripts/fix_gtfs.py --ingtf /vf/users/guibletwm/charlietest/charlie/ref/ref.gtf --outgtf /vf/users/guibletwm/charlietest/charlie/ref/ref.fixed.gtf
Traceback (most recent call last):
  File "/gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/CHARLIE/.v0.11.0/workflow/scripts/fix_gtfs.py", line 2, in <module>
    import pandas
  File "/data/CCBR_Pipeliner/Tools/ccbr_tools/v0.1/pandas/__init__.py", line 1
    from __future__ import annotations
    ^
SyntaxError: future feature annotations is not defined
kelly-sovacool commented 1 week ago

able to reproduce this with:

singularity shell -B /lscratch,/data/CCBR_Pipeliner,/data/sovacoolkl,/gpfs/gsfs10/users/CCBR_Pipeliner,/gpfs/gsfs12/users/sovacoolkl docker://nci
ccbr/charlie_star_ucsc_cufflinks:v0.4.0 zsh
python
import pandas
pandas

one workaround:

python -E
import pandas
pandas

we may want to use singularity -C or singularity -E, but will that mess up $SLURM_JOB_ID which is needed for temp dirs in some rules?

kelly-sovacool commented 1 week ago

when using singularity -C, $SLURM_JOB_ID is not available in the containers. However, snakemake calls singularity exec, which does not have the --env argument for setting the $SLURM_JOB_ID in the container. Also, it would likely use the value of $SLURM_JOB_ID from the master job instead of from within the individual rule jobs.

trying python -E inside the rule definition as a workaround.

kelly-sovacool commented 1 week ago

possible solutions: