violin_plot

[manarai]

Hi

I am running into the following errors. Seems that there is an incompatible enviromment which makes the plot_violine.R scripts fail. Could a --vanilla issue? Appreciate your help.

Error in rule violine_plots: jobid: 13 output: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata conda-env: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2

RuleException: CalledProcessError in line 32 of /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/rules/merge.smk: Command 'source activate /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2; set -euo pipefail; Rscript --vanilla /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/scripts/tmp0ihd5l4t.plot_violine.R ' returned non-zero exit status 1. File "/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/rules/merge.smk", line 32, in __rule_violine_plots File "/home/tommy/miniconda3/envs/dropseqpipe/lib/python3.6/concurrent/futures/thread.py", line 56, in run [Wed Jan 16 02:04:35 2019] Finished job 27. 35 of 38 steps (92%) done Shutting down, this might take some time.

[seb-mueller]

Hi manarai,

That's strange.

• What version are you using?
• How did you execute snakemake?

Also, could you try to execute the R-script manually? It's basically activating/running the environment and command mentioned above (you might have to run snakemake once again before):

source activate /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2
Rscript --vanilla /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/scripts/tmp0ihd5l4t.plot_violine.R

I'd be interested where the R-script brakes.

[manarai]

I am using conda 4.5.12 and the latest version of snakemake

tommy@d405:~$ source activate /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2 (/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2) tommy@d405:~$ Rscript --vanilla /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/scripts/tmp0ihd5l4t.plot_violine.R Fatal error: cannot open file '/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/scripts/tmp0ihd5l4t.plot_violine.R': No such file or directory

Then I try to run this way

(/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2) tommy@d405:~$ Rscript --vanilla /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/scripts/plot_violine.R

Attaching package: ‘cowplot’

The following object is masked from ‘package:ggplot2’:

ggsave

Error in dirname(mtx_path) : object 'snakemake' not found Calls: ReadMTX -> dirname Execution halted

[manarai]

I try to rerun it again and this is what it gives me.

snakemake --cores 8 --use-conda --directory ~/Nadia_projects/Nadia13/dropSeqPipe/ --rerun

.... /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//logs/bbmap/hipocampus_frozen_nuclei_hiseq_repair.txt contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake. Using shell: /bin/bash Provided cores: 8 Rules claiming more threads will be scaled down. Job counts: count jobs 1 all 1 violine_plots 2

[Thu Jan 17 02:56:36 2019] localrule violine_plots: input: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/umi/expression.mtx, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/read/expression.mtx, samples.csv output: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata jobid: 13 wildcards: results_dir=/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results/

Activating conda environment: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2

Attaching package: ‘cowplot’

The following object is masked from ‘package:ggplot2’:

ggsave

Error in grDevices::pdf(..., version = version) : cannot open file '/home/tommy/Nadia_projects/Nadia13/dropSeqPipe//home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf' Calls: ggsave -> do.call -> -> dev -> Execution halted [Thu Jan 17 02:56:54 2019] Error in rule violine_plots: jobid: 13 output: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata conda-env: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2

RuleException: CalledProcessError in line 32 of /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/rules/merge.smk: Command 'source activate /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2; set -euo pipefail; Rscript --vanilla /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/scripts/tmpeeuh4box.plot_violine.R ' returned non-zero exit status 1. File "/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/rules/merge.smk", line 32, in __rule_violine_plots File "/home/tommy/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/log/2019-01-17T025635.629628.snakemake.log

[manarai]

I try to rerun again and it stops when it runs the violin plot scripts

snakemake --cores 8 --use-conda --directory ~/Nadia_projects/Nadia13/dropSeqPipe/ --rerun

........ Provided cores: 8 Rules claiming more threads will be scaled down. Job counts: count jobs 1 all 1 violine_plots 2

[Thu Jan 17 02:56:36 2019] localrule violine_plots: input: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/umi/expression.mtx, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/read/expression.mtx, samples.csv output: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata jobid: 13 wildcards: results_dir=/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results/

Activating conda environment: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2

Attaching package: ‘cowplot’

The following object is masked from ‘package:ggplot2’:

ggsave

Error in grDevices::pdf(..., version = version) : cannot open file '/home/tommy/Nadia_projects/Nadia13/dropSeqPipe//home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf' Calls: ggsave -> do.call -> -> dev -> Execution halted [Thu Jan 17 02:56:54 2019] Error in rule violine_plots: jobid: 13 output: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata conda-env: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2

RuleException: CalledProcessError in line 32 of /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/rules/merge.smk: Command 'source activate /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2; set -euo pipefail; Rscript --vanilla /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/scripts/tmpeeuh4box.plot_violine.R ' returned non-zero exit status 1. File "/home/tommy/Nadia_projects/Nadia13/dropSeqPipe/rules/merge.smk", line 32, in __rule_violine_plots File "/home/tommy/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/log/2019-01-17T025635.629628.snakemake.log

[seb-mueller]

Thanks, that's useful information.

The culprit seems to be this line:

cannot open file '/home/tommy/Nadia_projects/Nadia13/dropSeqPipe//home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf'

It's doubling the path. I'd suspect this line to be the culprit:

https://github.com/Hoohm/dropSeqPipe/blob/48b0050c3e3627ff50f46e2951d05185f47e3fbe/scripts/plot_violine.R#L162

• I think the getwd() needs to be deleted. If you feel confident, could you try to delete from plot_violine.R file?

• Or, could you try to run it within the data directory (cd /home/tommy/Nadia_projects/Nadia13/dropSeqPipe) without the --directory parameter?

• By version earlier, I meant in fact the dropSeqPipe version? Ideally which commit you are on (e.g. running git show withing the git directory.

[manarai]

Right on, I deleted getwd() in the plot_violin.R scripts and it worked. Awesome! thank you very much.

snakemake --cores 8 --use-conda --directory ~/Nadia_projects/Nadia13/dropSeqPipe/ --rerun

Activating conda environment: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/conda/d01599d2

Attaching package: ‘cowplot’

The following object is masked from ‘package:ggplot2’:

ggsave

[Thu Jan 17 03:32:32 2019] Finished job 13. 1 of 2 steps (50%) done

[Thu Jan 17 03:32:32 2019] localrule all: input: /home/tommy/Nadia_projects/reference_nadia//mus_musculus_38_94/STAR_INDEX/SA_80/SA, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/fastqc_reads.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/fastqc_barcodes.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/adapter_content.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/barcode_filtering.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/RNA_filtering.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/knee_plots/hipocampus_frozen_nuclei_hiseq_knee_plot.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/star.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/yield.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/rna_metrics/hipocampus_frozen_nuclei_hiseq_rna_metrics.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/read/expression.mtx, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//samples/hipocampus_frozen_nuclei_hiseq/read/expression.mtx, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf, /home/tommy/Nadia_projects/reference_nadia//mus_musculus_38_94/STAR_INDEX/SA_80/SA, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/fastqc_reads.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/fastqc_barcodes.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/adapter_content.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/barcode_filtering.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/RNA_filtering.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/knee_plots/hipocampus_frozen_nuclei_hiseq_knee_plot.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//reports/star.html, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/yield.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/rna_metrics/hipocampus_frozen_nuclei_hiseq_rna_metrics.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/umi/expression.mtx, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//samples/hipocampus_frozen_nuclei_hiseq/umi/expression.mtx, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_counts.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/UMI_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/Count_vs_gene.pdf, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//summary/R_Seurat_objects.rdata, /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/results//plots/violinplots_comparison_UMI.pdf jobid: 0

[Thu Jan 17 03:32:32 2019] Finished job 0. 2 of 2 steps (100%) done Complete log: /home/tommy/Nadia_projects/Nadia13/dropSeqPipe/.snakemake/log/2019-01-17T033136.594431.snakemake.log

[seb-mueller]

Glad it worked and thanks for trying it out! I'll fix this soon and push it to the repository. Just out of interest, are you using Nadia from Dolomite Bio?

[manarai]

Thanks for your help. Yes I am using the Nadia system from Dolomite Bio.

On Jan 16, 2019, at 11:42, seb-mueller notifications@github.com wrote:

Glad it worked and thanks for trying it out! I'll fix this soon and push it to the repository. Just out of interest, are you using Nadia from Dolomite Bio?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/Hoohm/dropSeqPipe/issues/71#issuecomment-454892646, or mute the thread https://github.com/notifications/unsubscribe-auth/AmB6sPkWT7XePbEmlfHvxPukh4JFr4eWks5vD3KUgaJpZM4aBdIY.