Thank you very much for putting together this tool! I have been able to run dropSeqPipe on my local machine. However, I do not have enough memory for the alignment steps so I am now trying to run it in a Slurm-based cluster.
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.shbash Miniconda3-latest-Linux-x86_64.shgit clone https://github.com/Hoohm/dropSeqPipe.git conda create -n dropSeqPipe _(I need to run this because my cluster requires to create conda environments to install anything from bioconda. Otherwise I will get "Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from currentrepodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): | Killed")conda activate dropSeqPipeconda install -c bioconda -c conda-forge snakemake
I also set up a working directory containing the config.yaml, samples. csv, gtf_biotypes.yaml, custom_adapters.fa, cluster.yaml, results, data and temp-directory files and folders, respectively.
I then cd to the dropSeqPipe cloned git directory and run
This is what I get:
Building DAG of jobs... Creating conda environment ../dropSeqPipe/envs/merge_long.yaml... Downloading remote packages. Environment for ../dropSeqPipe/envs/merge_long.yaml created (location: .snakemake/conda/4aa40f8d) Creating conda environment ../dropSeqPipe/envs/bbmap.yaml... Downloading remote packages. Environment for ../dropSeqPipe/envs/bbmap.yaml created (location: .snakemake/conda/f3c61376) Creating conda environment ../dropSeqPipe/envs/umi_tools.yaml... Downloading remote packages. Environment for ../dropSeqPipe/envs/umi_tools.yaml created (location: .snakemake/conda/ff1b3e48) Creating conda environment ../dropSeqPipe/envs/r.yaml... Downloading remote packages. CreateCondaEnvironmentException: Could not create conda environment from /path/dropSeqPipe/rules/../envs/r.yaml: Collecting package metadata (repodata.json): ...working... done Solving environment: ...working...
I have tried both release/0.5 (code above) and release v0.4 .
The error I get with the latter is:
Building DAG of jobs... Creating conda environment ../dropSeqPipe-0.4/envs/merge_bam.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/merge_bam.yaml created (location: .snakemake/conda/c075d444) Creating conda environment https:/bitbucket.org/snakemake/snakemake-wrappers/raw/0.27.1/bio/fastqc/environment.yaml... Downloading remote packages. Environment for ../../../../../tmp/19332620.1.interactive/tmplqfz4fqe.yaml created (location: .snakemake/conda/d2883296) Creating conda environment https:/bitbucket.org/snakemake/snakemake-wrappers/raw/0.27.1/bio/star/align/environment.yaml... Downloading remote packages. Environment for ../../../../../tmp/19332620.1.interactive/tmp9zgsa5yw.yaml created (location: .snakemake/conda/65e76858) Creating conda environment ../dropSeqPipe-0.4/envs/star.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/star.yaml created (location: .snakemake/conda/389b25df) Creating conda environment ../dropSeqPipe-0.4/envs/umi_tools.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/umi_tools.yaml created (location: .snakemake/conda/ff1b3e48) Creating conda environment https:/bitbucket.org/snakemake/snakemake-wrappers/raw/0.21.0/bio/multiqc/environment.yaml... Downloading remote packages. Environment for ../../../../../tmp/19332620.1.interactive/tmpmng3qorb.yaml created (location: .snakemake/conda/e73a509a) Creating conda environment ../dropSeqPipe-0.4/envs/cutadapt.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/cutadapt.yaml created (location: .snakemake/conda/76021497) Creating conda environment ../dropSeqPipe-0.4/envs/dropseq_tools.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/dropseq_tools.yaml created (location: .snakemake/conda/0af7fa12) Creating conda environment ../dropSeqPipe-0.4/envs/picard.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/picard.yaml created (location: .snakemake/conda/26fdb8a2) Creating conda environment ../dropSeqPipe-0.4/envs/plots_ext.yaml... Downloading remote packages. CreateCondaEnvironmentException: Could not create conda environment from /path/dropSeqPipe-0.4/rules/../envs/plots_ext.yaml: Collecting package metadata (repodata.json): ...working... done Solving environment: ...working...
The fact that the error comes with r.yaml in one case and plots_ext.yaml in the other makes me think that there might be an issue with one of the R packages / versions when creating the environment.
If I try to run the recommended code from SLURM clusters , I get the following:
snakemake --cluster 'sbatch -n {cluster.n} -t {cluster.time} --clusters=nameofmycluster --output={cluster.output}' --jobs 1 --cluster-config /path/to/working_dir/cluster.yaml --use-conda --local-cores 1WorkflowError in line 20 of /path/to/cloned/github/repo/Snakefile: Config file config.yaml not found. File "/path/to/cloned/github/repo/Snakefile", line 20, in <module>
Just FYI, the recommended workflow in my cluster is to create a .condarc file in my personal folder, to use Anaconda3, and to create a conda environment. I have removed the .condarc file and am not using Anaconda3, since that also did not work. I am therefore using Miniconda 3 as recommended in the tutorials.
If I run echo $PATH I get:
/home/unix/myname/miniconda3/bin:/home/unix/myname/miniconda3/condabin:/home/unix/myname/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
Please let me know if you need any additional information. Thanks!
Hi,
Thank you very much for putting together this tool! I have been able to run dropSeqPipe on my local machine. However, I do not have enough memory for the alignment steps so I am now trying to run it in a Slurm-based cluster.
Below are the steps that I have taken (following these instructions):
Login and start interactive session.
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.shbash Miniconda3-latest-Linux-x86_64.shgit clone https://github.com/Hoohm/dropSeqPipe.gitconda create -n dropSeqPipe_(I need to run this because my cluster requires to create conda environments to install anything from bioconda. Otherwise I will get "Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: failed with repodata from currentrepodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): | Killed")conda activate dropSeqPipeconda install -c bioconda -c conda-forge snakemakeI also set up a working directory containing the config.yaml, samples. csv, gtf_biotypes.yaml, custom_adapters.fa, cluster.yaml, results, data and temp-directory files and folders, respectively.
I then cd to the dropSeqPipe cloned git directory and run
snakemake --use-conda --cores 1 --directory /path/to/my/working/directory
This is what I get:
Building DAG of jobs... Creating conda environment ../dropSeqPipe/envs/merge_long.yaml... Downloading remote packages. Environment for ../dropSeqPipe/envs/merge_long.yaml created (location: .snakemake/conda/4aa40f8d) Creating conda environment ../dropSeqPipe/envs/bbmap.yaml... Downloading remote packages. Environment for ../dropSeqPipe/envs/bbmap.yaml created (location: .snakemake/conda/f3c61376) Creating conda environment ../dropSeqPipe/envs/umi_tools.yaml... Downloading remote packages. Environment for ../dropSeqPipe/envs/umi_tools.yaml created (location: .snakemake/conda/ff1b3e48) Creating conda environment ../dropSeqPipe/envs/r.yaml... Downloading remote packages. CreateCondaEnvironmentException: Could not create conda environment from /path/dropSeqPipe/rules/../envs/r.yaml: Collecting package metadata (repodata.json): ...working... done Solving environment: ...working...I have tried both release/0.5 (code above) and release v0.4 .
The error I get with the latter is:
Building DAG of jobs... Creating conda environment ../dropSeqPipe-0.4/envs/merge_bam.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/merge_bam.yaml created (location: .snakemake/conda/c075d444) Creating conda environment https:/bitbucket.org/snakemake/snakemake-wrappers/raw/0.27.1/bio/fastqc/environment.yaml... Downloading remote packages. Environment for ../../../../../tmp/19332620.1.interactive/tmplqfz4fqe.yaml created (location: .snakemake/conda/d2883296) Creating conda environment https:/bitbucket.org/snakemake/snakemake-wrappers/raw/0.27.1/bio/star/align/environment.yaml... Downloading remote packages. Environment for ../../../../../tmp/19332620.1.interactive/tmp9zgsa5yw.yaml created (location: .snakemake/conda/65e76858) Creating conda environment ../dropSeqPipe-0.4/envs/star.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/star.yaml created (location: .snakemake/conda/389b25df) Creating conda environment ../dropSeqPipe-0.4/envs/umi_tools.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/umi_tools.yaml created (location: .snakemake/conda/ff1b3e48) Creating conda environment https:/bitbucket.org/snakemake/snakemake-wrappers/raw/0.21.0/bio/multiqc/environment.yaml... Downloading remote packages. Environment for ../../../../../tmp/19332620.1.interactive/tmpmng3qorb.yaml created (location: .snakemake/conda/e73a509a) Creating conda environment ../dropSeqPipe-0.4/envs/cutadapt.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/cutadapt.yaml created (location: .snakemake/conda/76021497) Creating conda environment ../dropSeqPipe-0.4/envs/dropseq_tools.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/dropseq_tools.yaml created (location: .snakemake/conda/0af7fa12) Creating conda environment ../dropSeqPipe-0.4/envs/picard.yaml... Downloading remote packages. Environment for ../dropSeqPipe-0.4/envs/picard.yaml created (location: .snakemake/conda/26fdb8a2) Creating conda environment ../dropSeqPipe-0.4/envs/plots_ext.yaml... Downloading remote packages. CreateCondaEnvironmentException: Could not create conda environment from /path/dropSeqPipe-0.4/rules/../envs/plots_ext.yaml: Collecting package metadata (repodata.json): ...working... done Solving environment: ...working...The fact that the error comes with r.yaml in one case and plots_ext.yaml in the other makes me think that there might be an issue with one of the R packages / versions when creating the environment.
If I try to run the recommended code from SLURM clusters , I get the following:
snakemake --cluster 'sbatch -n {cluster.n} -t {cluster.time} --clusters=nameofmycluster --output={cluster.output}' --jobs 1 --cluster-config /path/to/working_dir/cluster.yaml --use-conda --local-cores 1WorkflowError in line 20 of /path/to/cloned/github/repo/Snakefile: Config file config.yaml not found. File "/path/to/cloned/github/repo/Snakefile", line 20, in <module>Just FYI, the recommended workflow in my cluster is to create a .condarc file in my personal folder, to use Anaconda3, and to create a conda environment. I have removed the .condarc file and am not using Anaconda3, since that also did not work. I am therefore using Miniconda 3 as recommended in the tutorials.
If I run echo $PATH I get:
/home/unix/myname/miniconda3/bin:/home/unix/myname/miniconda3/condabin:/home/unix/myname/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/binPlease let me know if you need any additional information. Thanks!