search

CCBR / Pipeliner

An open-source and scalable solution to NGS analysis powered by the NIH's Biowulf cluster.
4 stars 0 forks source link

Python 2.7 module should be explicitly specified in all Pipeliner Rules. Python 2.7 will not be the default on biowulf. #440

Closed kopardev closed 4 years ago

kopardev commented 4 years ago

I found a few places where I may need to explicitly module load python/2.7 as to not cause any problems. I will update things as needed.

user@cn0949: $ find . -type f -exec grep 'module' {} \+ | grep 'python' | grep -v 'python/' | grep 'python'
./Rules/ChIPseq.snakefile:module load {params.pythonver}
./Rules/InitialChIPseqQC.snakefile:        commoncmd="module load {params.deeptoolsver}; module load python;"
./Rules/InitialChIPseqQC.snakefile:        commoncmd="module load {params.deeptoolsver}; module load python;"
./Rules/InitialChIPseqQC.snakefile:        commoncmd="module load {params.deeptoolsver}; module load python;"
./Rules/InitialChIPseqQC.snakefile:        commoncmd="module load {params.deeptoolsver}; module load python;"
./Rules/initialqcrnaseq.snakefile:    os.system("module load "+params.pythonver+";python "+params.pythonscript+" "+input.groupsfile+" > "+output.outfile)
./Rules/initialqcrnaseq.snakefile:module load {params.pythonver}
./Rules/rnaseq.snakefile:module load {params.pythonver}
./Rules/rnaseq.snakefile:module load {params.pythonver}
./Rules/rnaseq.snakefile:module load {params.pythonver}
./Rules/rnaseqforfusions.rl:  shell: "module load R/3.5;module load picard/2.17.11; java -Xmx10g -jar $PICARDJARPATH/picard.jar CollectRnaSeqMetrics REF_FLAT={params.refflat} INPUT={input.file1} OUTPUT={output.outstar1} RIBOSOMAL_INTERVALS={params.rrnalist}  STRAND_SPECIFICITY=SECOND_READ_TRANSCRIPTION_STRAND TMP_DIR=/lscratch/$SLURM_JOBID  VALIDATION_STRINGENCY=SILENT; java -Xmx10g -jar $PICARDJARPATH/picard.jar CollectInsertSizeMetrics INPUT={input.file1} OUTPUT={output.outstar3} HISTOGRAM_FILE={output.outstar4} MINIMUM_PCT=0.5 TMP_DIR=/lscratch/$SLURM_JOBID ;module load samtools; samtools flagstat {input.file1} > {output.outstar2}; module load python; python Scripts/bam_count_concord_stats.py {input.file1} >> {output.outstar2} "
./Rules/snpeff.new.rl:          shell("module load snpEff;{params.snpeff} -v {params.genome} -c {params.effconfig} %s -interval {params.hapmap} interval {params.regulomeDB} -stats %s > %s; cat %s | python {params.firstanno}> %s; cat %s | {params.snpsift} filter \"{params.filtersift}\" > %s"%(ifile,ohtml,osnpeff,osnpeff,ofirst,ofirst,ofilter))      
./Rules/snpeff.new.rl:#         shell("module load snpEff; {params.snpeff} -v {params.genome} -c {params.effconfig} {0} -stats {1} > {2}; cat {2} | python {params.firstanno}> {3}; cat {3} | {params.snpsift} filter \"{params.filtersift}\" > {4}".format(ifile,ohtml,osnpeff,ofirst,ofilter))
./Rules/strelka.rl:    shell: "module load strelka/2.9.0; module load python; configureStrelkaSomaticWorkflow.py --ref={params.genome} --tumor={params.dir}/{input.tumor} --normal={params.dir}/{input.normal} --runDir={output.outdir} --exome; cd {output.outdir}; ./runWorkflow.py -m local -j {threads}; {params.gatk} -T CombineVariants -R {params.genome} --variant results/variants/somatic.snvs.vcf.gz --variant results/variants/somatic.indels.vcf.gz --assumeIdenticalSamples --filteredrecordsmergetype KEEP_UNCONDITIONAL -o {output.vcf}; {params.gatk} -T SelectVariants -R {params.genome} --variant {output.vcf} --excludeFiltered -o {output.vcffiltered}; cd {params.dir}/strelka_out; module load samtools/1.6; echo $'NORMAL {params.normalsample}\nTUMOR {params.tumorsample}' > {output.names}; bcftools reheader -o {output.final} -s {output.names} {output.vcffiltered}; module load snpEff/4.3t; java -Xmx12g -jar $SNPEFF_JAR -v {params.snpeffgenome} -c {params.effconfig} -cancer -canon -csvStats {output.csvstats} -stats {output.htmlstats} -cancerSamples {params.dir}/pairs {output.final} > {output.out}"
./Rules/wgs_strelka.rl:    shell: "module load strelka/2.9.0; module load python; configureStrelkaSomaticWorkflow.py --ref={params.genome} --tumor={params.dir}/{input.tumor} --normal={params.dir}/{input.normal} --runDir={output.outdir}; cd {output.outdir}; ./runWorkflow.py -m local -j {threads}; {params.gatk} -T CombineVariants -R {params.genome} --variant results/variants/somatic.snvs.vcf.gz --variant results/variants/somatic.indels.vcf.gz --assumeIdenticalSamples --filteredrecordsmergetype KEEP_UNCONDITIONAL -o {output.vcf}; {params.gatk} -T SelectVariants -R {params.genome} --variant {output.vcf} --excludeFiltered -o {output.vcffiltered}; cd {params.dir}/strelka_out; module load samtools/1.6; echo $'NORMAL {params.normalsample}\nTUMOR {params.tumorsample}' > {output.names}; bcftools reheader -o {output.final} -s {output.names} {output.vcffiltered}; module load snpEff/4.3t; java -Xmx12g -jar $SNPEFF_JAR -v {params.snpeffgenome} -c {params.effconfig} -cancer -canon -csvStats {output.csvstats} -stats {output.htmlstats} -cancerSamples {params.dir}/pairs {output.final} > {output.out}"

In reponse to: 

Dear HPC users,

Support for Python 2.7 has officially ended January 1, 2020 though one more release is planned for mid April 2020. See

            https://www.python.org/dev/peps/pep-0373/
            https://www.python.org/doc/sunset-python-2/

Therefore, on April 15, 2020 the python/3.7 module will become the default Python module on biowulf. The python/2.7 module will continue to be available after this date but not as the default.

Please update your code and workflows as needed.
skchronicles commented 4 years ago

Resolved

Explicitly loading exact module of python as needed