search

maxplanck-ie / snakepipes

Customizable workflows based on snakemake and python for the analysis of NGS data
http://snakepipes.readthedocs.io
381 stars 85 forks source link

Error when running ATAC-seq workflow: 1. Not a conda environment; 2. /usr/bin/bash: plotCoverage command not found; #765

Closed hero-outman closed 3 years ago

hero-outman commented 3 years ago

Hello dear fellows,

I met an error when I am running snakepipes ATAC_seq workflow, based on the log file it's like:

  1. Can not activating conda environment /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d, Not a conda environment;
  2. Tools from deeptools like plotCoverage, command not found;

I will describe details and post part of the log file below:

I use snakepipes on my local centos7 server(not a cluster). After created envs and set up the indexes, I use some exsited filtered bam as inpput to run ATAC_seq pipeline and got errors. code:

ATAC-seq -d $WORKINGDIR \
    --local \
    -j 40 \
    --DAG \
    --peakCaller MACS2 \
    --sampleSheet sampleSheet.tsv \
    --fromBAM $filterbam \
    --bamExt .markdup.bam \
    hg38 &> snake.atac.log

part of the log file:

Sample sheet found and header is ok!

---- This analysis has been done using snakePipes version 2.5.0 ----
Sample sheet found and header is ok!
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 40
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   CSAW
    1   CSAW_report
    4   MACS2_peak_qc
    1   all
    4   bamCompare_subtract
    4   bamCoverage_filtered
    1   bamPE_fragment_size
    3   calc_matrix_cov_CSAW
    4   callOpenChromatin
    4   estimate_read_filtering
    4   filterCoveragePerScaffolds
    4   filterFragments
    1   filterMetricsToHtml
    3   get_nearest_gene
    3   get_nearest_transcript
    1   multiBamSummary
    1   plotCorrelation_pearson
    1   plotCorrelation_spearman
    1   plotCoverage
    1   plotFingerprint
    1   plotPCA
    3   plot_heatmap_cov_CSAW
    51

Activating conda environment: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d
Activating conda environment: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d
Not a conda environment: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d

[Mon May 17 11:22:34 2021]
Error in rule plotCoverage:
    jobid: 31
    output: deepTools_qc/plotCoverage/read_coverage.tsv
    log: deepTools_qc/logs/plotCoverage.out, deepTools_qc/logs/plotCoverage.err (check log file(s) for error message)
    conda-env: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d
    shell:

    plotCoverage ...some_params... Coverage.out 2> deepTools_qc/logs/plotCoverage.err

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Not a conda environment: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d

Job failed, going on with independent jobs.

...more conda environment issue...
Activating conda environment: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d
... ...
Not a conda environment: /opt/anaconda/miniconda3/envs/37c86fa5b74354c13ff20f5966bcf03d

...more job failed issue...

Exiting because a job execution failed. Look above for error message
Complete log: path_to_log/log/2021-05-17T112231.671709.snakemake.log

 !!! ERROR in the ATAC-seq open chromatin workflow! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Error: snakemake returned an error code of 1, so processing is incomplete!

check logs under deepTools_qc/logs/ folder, said:

/usr/bin/bash: xxx: command not found

I think it's some env issue caused snakepipes cannot find the path of deeptools, I try to run snakePipes createEnvs again, it showed a list and stoped:

Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/shared.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/createIndices.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/rna_seq.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/rMats.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/sc_rna_seq.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/sc_rna_seq_seurat3.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/sc_rna_seq_loompy.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/sc_rna_seq_alevinqc.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/sc_rna_seq_eisaR.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/dna_mapping.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/chip_seq.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/atac_seq.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/hic.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/wgbs.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/rmarkdown.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/preprocessing.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/noncoding.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/sambamba.yaml in /opt/anaconda/miniconda3/envs
Creating environment from /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/pysam.yaml in /opt/anaconda/miniconda3/envs

I have no more clue. Can I get some hint on getting rid of this error?

LeilyR commented 3 years ago

Set conda-prefix to where your envs actuallly are. This would solve your issue. Try to set it using --snakemakeOptions.

hero-outman commented 3 years ago

Hi @LeilyR Thank you so much for the reply! I think I should add --snakemakeOptions='--conda-prefix my_envs_path' in command. But not sure which one to set as my_envs_path : /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/ or /opt/anaconda/miniconda3/envs Could you help me more to figure it out?

hero-outman commented 3 years ago

Thank you LeilyR, with --snakemakeOptions='--conda-prefix /myhome/.conda/envs/snakePipes/lib/python3.9/site-packages/snakePipes/shared/rules/envs/' added as param, I got workflow running and no more issue. Thank you so much!