search

maxplanck-ie / snakepipes

Customizable workflows based on snakemake and python for the analysis of NGS data
http://snakepipes.readthedocs.io
381 stars 85 forks source link

after snakePipes createEnvs, createIndices still re-create env #665

Closed yxian9 closed 4 years ago

yxian9 commented 4 years ago

Hi, thanks for providing such great tools.

I followed the tutorial. installed using conda create -n snakePipes -c mpi-ie -c conda-forge -c bioconda snakePipes==2.1.1 conda activate snakePipes snakePipes createEnvs, then create index using createIndices --tools hisat2 bowtie2 -o GRCh37_v75 -j 12 --genome ftp://ftp.ensembl.org/pub/release75/fasta/homo_sapiens/dna/Homo_sapiens.GRCh37.75.dna.primary_assembly.fa.gz --gtf ftp://ftp.ensembl.org/pub/release-75/gtf/homo_sapiens/Homo_sapiens.GRCh37.75.gtf.gz GRCh37_v75

However, createIndices cmd still create required env.

---- This analysis has been done using snakePipes version 2.1.1 ---- Building DAG of jobs... Creating conda environment /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/createIndices.yaml... Downloading and installing remote packages. Environment for ../../../../../../../data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/createIndices.yaml created (location: ../../../../../../../data/diaolab/software/miniconda3/envs/6a06a454) Creating conda environment /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/shared.yaml... Downloading and installing remote packages. Environment for ../../../../../../../data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/shared.yaml created (location: ../../../../../../../data/diaolab/software/miniconda3/envs/d730d58b) Creating conda environment /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/dna_mapping.yaml... Downloading and installing remote packages.

It looks like the envs created by snakePipes createEnvs is not detected.

Here is the software version I used

>conda -V
conda 4.8.3
>conda list snakePipes
# packages in environment at /data/diaolab/software/miniconda3/envs/snakePipes:
# Name                    Version                   Build  Channel
snakepipes                2.1.1                      py_0    mpi-ie
>conda list snakemake
# packages in environment at /data/diaolab/software/miniconda3/envs/snakePipes:
# Name                    Version                   Build  Channel
snakemake                 5.20.1                        0    bioconda
snakemake-minimal         5.20.1                     py_0    bioconda

to confirm snakePipes createEnvs works. I re-run snakePipes createEnvs and get follow message

Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/shared.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/createIndices.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/rna_seq.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/sc_rna_seq.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/sc_rna_seq_seurat3.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/sc_rna_seq_loompy.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/dna_mapping.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/chip_seq.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/histone_hmm.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/atac_seq.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/hic.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/wgbs.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/rmarkdown.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/preprocessing.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/noncoding.yaml in /data/diaolab/software/miniconda3/envs
Creating environment from /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/rules/envs/sambamba.yaml in /data/diaolab/software/miniconda3/envs

confirmed the env in conda envs directory

ls  -1 /data/diaolab/software/miniconda3/envs
1ce4fe3ced6c0d80ada6f6033c100cc1
1e5a08451310eb68e6eea88366cc0b1a
285a0786eeb2b0dfd1caba5ef18aea69
4be49c95d2b8109a9f453ee07ba99d22
51271ba56bf1d596edcfc98e036ffed5
53faa5d712f95e61365bcc8c4316cc63
582369916bebd320d423cdfa12ce9cee
6a06a454
6a06a454.yaml
7098094a54af801b7583730797b07902
73ad4c623c82c19561e0dc19f4a0695b
81bc68c6083359bdd77df4a15ff05b9a
82cf6747
82cf6747.yaml
a3ee79fe3aa3f3beb3dd1ce7c6ea8750
a961d32d68918fec99f58d6305933fc8
aace6f24f2dedad7e8e58f448a99ccfc
cb94f6ca2a041af6d69a56a625d06431
d730d58b
d730d58b.yaml
d73112a88c7706706e942034c0177072
db470627
db470627.yaml
de5d29ec5f715233f33fe78e5b3fa393
py27
snakePipes
test

the 6a06a454 and 6a06a454.yaml are created by the createIndices cmd.

I also confirmed the shared/defaults.yaml. the snakemakeOptions is not changed.

 cat /data/diaolab/software/miniconda3/envs/snakePipes/lib/python3.8/site-packages/snakePipes/shared/defaults.yaml
clusterConfig: shared/cluster.yaml
configMode: manual
emailSender: null
oldConfig: null
onlySSL: false
organismsDir: shared/organisms
smtpPassword: null
smtpPort: 0
smtpServer: null
smtpUsername: null
snakemakeOptions: ' --use-conda --conda-prefix /data/diaolab/software/miniconda3/envs '
tempDir: /data/extended/
toolsVersion: true

Would you please advice how to fix this ?

by the way, I tried the old version of snakePipes using conda create -n snakePipes -c mpi-ie -c conda-forge -c bioconda snakePipes . Everything works good.

Thank you for your help!

yxian9 commented 4 years ago

the envs created by the sub-cmd, such as 'createEnvs' or 'DNA-mapping' can be recognized by the whole workflow. So in my case, the envs will need to be created twice, then everything works fine. Not a big deal.