search

gagneurlab / drop

Pipeline to find aberrant events in RNA-Seq data, useful for diagnosis of rare disorders
MIT License
133 stars 44 forks source link

DROP error when using own data #235

Closed svergult closed 2 years ago

svergult commented 3 years ago

Hi

We installed DROP on our HPC and tried to run the demo data via drop demo, which worked fine.

However, we are now trying to run DROP on our own data and seem to get some errors. I hope you could give us some tips and tricks. Thanks!

drop sampleAnnotation does not give any error.

when trying:

snakemake aberrantExpression --cores 10

we get the following errors (I pasted the log files)


WARNING: Less than 30 IDs in DROP_GROUP outrider
WARNING: Less than 30 IDs in DROP_GROUP fraser
check for missing R packages
Structuring dependencies...
Dependencies file generated at: /tmp/tmpqkwta24q

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   AberrantExpression_pipeline_Counting_Datasets_R
    1   AberrantExpression_pipeline_Counting_Summary_R
    10  AberrantExpression_pipeline_Counting_countReads_R
    1   AberrantExpression_pipeline_Counting_filterCounts_R
    1   AberrantExpression_pipeline_Counting_mergeCounts_R
    1   AberrantExpression_pipeline_OUTRIDER_Datasets_R
    1   AberrantExpression_pipeline_OUTRIDER_Summary_R
    1   AberrantExpression_pipeline_OUTRIDER_results_R
    1   AberrantExpression_pipeline_OUTRIDER_runOutrider_R
    1   aberrantExpression
    1   aberrantExpression_dependency
    20

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000006_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000006.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000006.Rds
    jobid: 7
    wildcards: annotation=v38, sampleID=RN2000006

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010122_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010122.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010122.Rds
    jobid: 8
    wildcards: annotation=v38, sampleID=RNA010122

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000004_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000004.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000004.Rds
    jobid: 12
    wildcards: annotation=v38, sampleID=RN2000004

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000005_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000005.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000005.Rds
    jobid: 5
    wildcards: annotation=v38, sampleID=RN2000005

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010356_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010356.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010356.Rds
    jobid: 13
    wildcards: annotation=v38, sampleID=RNA010356

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000001_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000001.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000001.Rds
    jobid: 11
    wildcards: annotation=v38, sampleID=RN2000001

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010120_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010120.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010120.Rds
    jobid: 14
    wildcards: annotation=v38, sampleID=RNA010120

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010118_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010118.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010118.Rds
    jobid: 9
    wildcards: annotation=v38, sampleID=RNA010118

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RN2000008_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000008.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000008.Rds
    jobid: 15
    wildcards: annotation=v38, sampleID=RN2000008

[Mon Jul 26 17:28:37 2021]
rule AberrantExpression_pipeline_Counting_countReads_R:
    input: /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run084/Alignment/output/RNA010559_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam, /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/count_ranges.Rds, Scripts/AberrantExpression/pipeline/Counting/countReads.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010559.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010559.Rds
    jobid: 10
    wildcards: annotation=v38, sampleID=RNA010559

[Mon Jul 26 17:29:10 2021]
[Mon Jul 26 17:29:10 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 7
    jobid: 10
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000006.Rds
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010559.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000006.Rds (check log file(s) for error message)
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010559.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmptk1080eb.countReads.R' returned non-zero exit status 137.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmp8gk5eieh.countReads.R' returned non-zero exit status 135.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:29:10 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 15
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000008.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000008.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpya3u9gun.countReads.R' returned non-zero exit status 135.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:29:10 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 11
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000001.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000001.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmp32fh_61l.countReads.R' returned non-zero exit status 139.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:29:10 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 13
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010356.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010356.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpz_g6_ubp.countReads.R' returned non-zero exit status 135.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:29:10 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 9
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010118.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010118.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpwcw84vec.countReads.R' returned non-zero exit status 135.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Terminating processes on user request, this might take some time.
[Mon Jul 26 17:30:07 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 5
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000005.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000005.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmp6wmpeyuj.countReads.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:30:08 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 12
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RN2000004.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RN2000004.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpyz1dvxgj.countReads.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:30:09 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 14
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010120.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010120.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpwgjlssly.countReads.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:30:11 2021]
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
    jobid: 8
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_expression/v38/counts/RNA010122.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AE/v38/counts/RNA010122.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 34 of /tmp/tmpqkwta24q:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpdngsjt8c.countReads.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpqkwta24q", line 34, in __rule_AberrantExpression_pipeline_Counting_countReads_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Complete log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/log/2021-07-26T172833.439259.snakemake.log

when trying:

snakemake mae --cores 10

we get the following errors


WARNING: Less than 30 IDs in DROP_GROUP outrider
WARNING: Less than 30 IDs in DROP_GROUP fraser
check for missing R packages
Structuring dependencies...
Dependencies file generated at: /tmp/tmpk7rioi1v

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 12
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   MonoallelicExpression_pipeline_MAE_Datasets_R
    1   MonoallelicExpression_pipeline_MAE_Results_R
    1   MonoallelicExpression_pipeline_MAE_deseq_mae_R
    1   MonoallelicExpression_pipeline_QC_DNA_RNA_matrix_plot_R
    1   MonoallelicExpression_pipeline_QC_Datasets_R
    1   MonoallelicExpression_pipeline_QC_create_matrix_dna_rna_cor_R
    1   MonoallelicExpression_pipeline_QC_deseq_qc_R
    1   mae
    8

[Mon Jul 26 17:43:31 2021]
rule MonoallelicExpression_pipeline_MAE_deseq_mae_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/D1913781--RNA010559.csv.gz, Scripts/MonoallelicExpression/pipeline/MAE/deseq_mae.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_results/mae/samples/D1913781--RNA010559_res.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/D1913781--RNA010559.Rds
    jobid: 3
    wildcards: vcf=D1913781, rna=RNA010559

[Mon Jul 26 17:43:31 2021]
rule MonoallelicExpression_pipeline_QC_deseq_qc_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/qc_RNA010559.csv.gz, Scripts/MonoallelicExpression/pipeline/QC/deseq_qc.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/RNA_GT/RNA010559.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/qc_RNA010559.Rds
    jobid: 12
    wildcards: rna=RNA010559

[Mon Jul 26 17:43:45 2021]
Error in rule MonoallelicExpression_pipeline_QC_deseq_qc_R:
    jobid: 12
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/RNA_GT/RNA010559.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/qc_RNA010559.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 321 of /tmp/tmpk7rioi1v:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpqfj7wrax.deseq_qc.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpk7rioi1v", line 321, in __rule_MonoallelicExpression_pipeline_QC_deseq_qc_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 17:43:45 2021]
Error in rule MonoallelicExpression_pipeline_MAE_deseq_mae_R:
    jobid: 3
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_results/mae/samples/D1913781--RNA010559_res.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/D1913781--RNA010559.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 284 of /tmp/tmpk7rioi1v:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpxhf24cl6.deseq_mae.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpk7rioi1v", line 284, in __rule_MonoallelicExpression_pipeline_MAE_deseq_mae_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/log/2021-07-26T174327.541662.snakemake.log

and when trying

snakemake aberrantSplicing --cores 10

we get the following errors


WARNING: Less than 30 IDs in DROP_GROUP outrider
WARNING: Less than 30 IDs in DROP_GROUP fraser
check for missing R packages
Structuring dependencies...
Dependencies file generated at: /tmp/tmpketh60we

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    10  AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
    1   AberrantSplicing_pipeline_Counting_01_2_countRNA_splitReads_merge_R
    10  AberrantSplicing_pipeline_Counting_01_3_countRNA_nonSplitReads_samplewise_R
    1   AberrantSplicing_pipeline_Counting_01_4_countRNA_nonSplitReads_merge_R
    1   AberrantSplicing_pipeline_Counting_01_5_countRNA_collect_R
    1   AberrantSplicing_pipeline_Counting_02_psi_value_calculation_FraseR_R
    1   AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R
    1   AberrantSplicing_pipeline_Counting_DatasetsF_R
    1   AberrantSplicing_pipeline_Counting_Summary_R
    1   AberrantSplicing_pipeline_FRASER_04_fit_hyperparameters_FraseR_R
    1   AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R
    1   AberrantSplicing_pipeline_FRASER_06_calculation_stats_AE_FraseR_R
    1   AberrantSplicing_pipeline_FRASER_07_extract_results_FraseR_R
    1   AberrantSplicing_pipeline_FRASER_Datasets_R
    1   AberrantSplicing_pipeline_FRASER_Summary_R
    1   aberrantSplicing
    34

[Mon Jul 26 18:00:51 2021]
rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/sample_tmp/splitCounts/sample_RNA010559.done
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AS/fraser/splitReads/RNA010559.Rds
    jobid: 18
    wildcards: dataset=fraser, sample_id=RNA010559
    threads: 3

[Mon Jul 26 18:00:51 2021]
rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/sample_tmp/splitCounts/sample_RNA010120.done
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AS/fraser/splitReads/RNA010120.Rds
    jobid: 7
    wildcards: dataset=fraser, sample_id=RNA010120
    threads: 3

[Mon Jul 26 18:00:51 2021]
rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/sample_tmp/splitCounts/sample_RN2000004.done
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AS/fraser/splitReads/RN2000004.Rds
    jobid: 16
    wildcards: dataset=fraser, sample_id=RN2000004
    threads: 3

[Mon Jul 26 18:01:33 2021]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 18
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/sample_tmp/splitCounts/sample_RNA010559.done
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AS/fraser/splitReads/RNA010559.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 128 of /tmp/tmpketh60we:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmp9f25c2ou.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpketh60we", line 128, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 18:01:33 2021]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 16
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/sample_tmp/splitCounts/sample_RN2000004.done
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AS/fraser/splitReads/RN2000004.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 128 of /tmp/tmpketh60we:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpi741ny56.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpketh60we", line 128, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
[Mon Jul 26 18:01:33 2021]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 7
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/aberrant_splicing/datasets/cache/raw-fraser/sample_tmp/splitCounts/sample_RNA010120.done
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/AS/fraser/splitReads/RNA010120.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 128 of /tmp/tmpketh60we:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpp6jy5cai.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmpketh60we", line 128, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/log/2021-07-26T180047.733743.snakemake.log

Below you can find the sample annotation file (10 RNA-seq files, 1 vcf file) as well as the config file. Thanks!

Sample Annotation file

RNA_ID | RNA_BAM_FILE | DROP_GROUP | PAIRED_END | COUNT_MODE | COUNT_OVERLAPS | DNA_VCF_FILE | DNA_ID | STRAND | HPO_TERMS | GENE_COUNTS_FILE | ANNOTATION -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- RNA010559 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run084/Alignment/output/RNA010559_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser,mae | TRUE | IntersectionStrict | TRUE | /data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Proband_11_01472-gatk-haplotype-joint-annotated-decomposed.vcf | D1913781 | no |   |   |   RN2000001 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000001_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RN2000004 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000004_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RN2000005 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000005_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RN2000006 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/WES_Negatieve_Stalen/Alignment/output/RN2000006_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RN2000008 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RN2000008_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RNA010118 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010118_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RNA010120 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010120_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RNA010122 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010122_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   RNA010356 | /arcanine/scratch/gent/vo/001/gvo00101/projects/RNA-Sequencing_in_the_Quest_for_Rare_Diseases/NVQ_Run106/Controle_Stalen/Alignment/output/RNA010356_rRNA_Filtered_STAR_twoPass_Aligned.sortedByCoord.out.bam | outrider,fraser | TRUE | IntersectionStrict | TRUE |   |   | no |   |   |   **Config file** ``` projectTitle: Detection of RNA Outlier Pipeline htmlOutputPath: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/html indexWithFolderName: true root: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output sampleAnnotation: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/sample_annotation_ownsamples.tsv geneAnnotation: v38: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/Data/gencode.v38.annotation.gtf genomeAssembly: hg38 #genome: /kyukon/data/gent/vo/001/gvo00101/genomes/Hsapiens/hg38/seq/hg38.fa #genome: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/drop_demo/Data/chr21.fa hpoFile: null random_seed: true exportCounts: geneAnnotations: - v38 #excludeGroups: # - mae aberrantExpression: groups: - outrider fpkmCutoff: 1 implementation: autoencoder padjCutoff: 1 zScoreCutoff: 0 maxTestedDimensionProportion: 3 dassie: tssWindow: 500 pasWindow: 1000 aberrantSplicing: groups: - fraser recount: true longRead: false keepNonStandardChrs: true filter: false minExpressionInOneSample: 20 minDeltaPsi: 0.05 implementation: PCA padjCutoff: 1 zScoreCutoff: 0 deltaPsiCutoff: 0.05 maxTestedDimensionProportion: 6 mae: groups: - mae genome: /data/gent/vo/001/gvo00101/genomes/Hsapiens/hg38/seq/hg38.fa # path to genome sequence in fasta format gatkIgnoreHeaderCheck: true padjCutoff: .05 allelicRatioCutoff: 0.8 addAF: false maxAF: .001 maxVarFreqCohort: .04 #standard .05 #VCF-BAM matching qcVcf: /data/gent/vo/001/gvo00101/vsc42523/DROP/Data/qc_vcf_1000G_hg38.vcf.gz # path to common variant file e.g. qc_vcf_1000G.vcf.gz qcGroups: mae tools: gatkCmd: gatk bcftoolsCmd: bcftools samtoolsCmd: samtools ```
vyepez88 commented 3 years ago

Hi, the error Command 'set -euo pipefail; Rscript --vanilla usually means that there was insufficient memory. Can you please confirm that the cluster has enough memory to run the different BAM files in parallel (~15Gb per BAM file)?

svergult commented 3 years ago

Thanks, this was indeed the problem. The aberrantExpression and aberrantSplicing module now work.

However, for the MAE module, I got the following error, see below. Does this still have to do with insufficient memory? It also mentions " counts matrix should be numeric, currently it has mode: logical".

WARNING: GATK v4.2.0.0 support for Java 11 is in beta state. Use at your own risk.
WARNING: Less than 30 IDs in DROP_GROUP outrider
WARNING: Less than 30 IDs in DROP_GROUP fraser
check for missing R packages
Structuring dependencies...
Dependencies file generated at: /tmp/tmp_sxafshg

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 20
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   MonoallelicExpression_pipeline_MAE_Datasets_R
    1   MonoallelicExpression_pipeline_MAE_Results_R
    1   MonoallelicExpression_pipeline_MAE_deseq_mae_R
    1   MonoallelicExpression_pipeline_QC_DNA_RNA_matrix_plot_R
    1   MonoallelicExpression_pipeline_QC_Datasets_R
    1   MonoallelicExpression_pipeline_QC_create_matrix_dna_rna_cor_R
    1   MonoallelicExpression_pipeline_QC_deseq_qc_R
    1   mae
    8

[Thu Jul 29 22:14:00 2021]
rule MonoallelicExpression_pipeline_MAE_deseq_mae_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/D1913781--RNA010559.csv.gz, Scripts/MonoallelicExpression/pipeline/MAE/deseq_mae.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_results/mae/samples/D1913781--RNA010559_res.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/D1913781--RNA010559.Rds
    jobid: 3
    wildcards: vcf=D1913781, rna=RNA010559

[Thu Jul 29 22:14:00 2021]
rule MonoallelicExpression_pipeline_QC_deseq_qc_R:
    input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/qc_RNA010559.csv.gz, Scripts/MonoallelicExpression/pipeline/QC/deseq_qc.R
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/RNA_GT/RNA010559.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/qc_RNA010559.Rds
    jobid: 12
    wildcards: rna=RNA010559

Started with deseq
Error in DESeqDataSet(se, design = design, ignoreRank) : 
  counts matrix should be numeric, currently it has mode: logical
Calls: DESeq4MAE ... deseq_for_allele_specific_expression -> DESeqDataSetFromMatrix -> DESeqDataSet
Execution halted
[Thu Jul 29 22:14:12 2021]
Error in rule MonoallelicExpression_pipeline_QC_deseq_qc_R:
    jobid: 12
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/RNA_GT/RNA010559.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/qc_RNA010559.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 321 of /tmp/tmp_sxafshg:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmp9ko8y51q.deseq_qc.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmp_sxafshg", line 321, in __rule_MonoallelicExpression_pipeline_QC_deseq_qc_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Error in DESeqDataSet(se, design = design, ignoreRank) : 
  counts matrix should be numeric, currently it has mode: logical
Calls: DESeq4MAE ... deseq_for_allele_specific_expression -> DESeqDataSetFromMatrix -> DESeqDataSet
Execution halted
[Thu Jul 29 22:14:13 2021]
Error in rule MonoallelicExpression_pipeline_MAE_deseq_mae_R:
    jobid: 3
    output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_results/mae/samples/D1913781--RNA010559_res.Rds
    log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/D1913781--RNA010559.Rds (check log file(s) for error message)

RuleException:
CalledProcessError in line 284 of /tmp/tmp_sxafshg:
Command 'set -euo pipefail;  Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpiwpz50ql.deseq_mae.R' returned non-zero exit status 1.
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/tmp/tmp_sxafshg", line 284, in __rule_MonoallelicExpression_pipeline_MAE_deseq_mae_R
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/log/2021-07-29T221349.822586.snakemake.log
nickhsmith commented 3 years ago

Could you please make sure that you aren't overloading your system? You are asking it to run on 20 cores. Could you please run it with less, and also add the verbose tag.

snakemake --cores 5 --verbose

Can you also confirm that your input csv file has formed correctly by showing me the results of this command?

zcat /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/D1913781--RNA010559.csv.gz | head

Thanks a lot for using DROP, I hope that we can get this solved for you soon.

On Fri, Jul 30, 2021 at 8:45 AM svergult @.***> wrote:

Thanks, this was indeed the problem. The aberrantExpression and aberrantSplicing module now work.

However, for the MAE module, I got the following error, see below. Does this still have to do with insufficient memory? It also mentions " counts matrix should be numeric, currently it has mode: logical".

WARNING: GATK v4.2.0.0 support for Java 11 is in beta state. Use at your own risk. WARNING: Less than 30 IDs in DROP_GROUP outrider WARNING: Less than 30 IDs in DROP_GROUP fraser check for missing R packages Structuring dependencies... Dependencies file generated at: /tmp/tmp_sxafshg

Building DAG of jobs... Using shell: /bin/bash Provided cores: 20 Rules claiming more threads will be scaled down. Job counts: count jobs 1 MonoallelicExpression_pipeline_MAE_Datasets_R 1 MonoallelicExpression_pipeline_MAE_Results_R 1 MonoallelicExpression_pipeline_MAE_deseq_mae_R 1 MonoallelicExpression_pipeline_QC_DNA_RNA_matrix_plot_R 1 MonoallelicExpression_pipeline_QC_Datasets_R 1 MonoallelicExpression_pipeline_QC_create_matrix_dna_rna_cor_R 1 MonoallelicExpression_pipeline_QC_deseq_qc_R 1 mae 8

[Thu Jul 29 22:14:00 2021] rule MonoallelicExpression_pipeline_MAE_deseq_mae_R: input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/D1913781--RNA010559.csv.gz, Scripts/MonoallelicExpression/pipeline/MAE/deseq_mae.R output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_results/mae/samples/D1913781--RNA010559_res.Rds log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/D1913781--RNA010559.Rds jobid: 3 wildcards: vcf=D1913781, rna=RNA010559

[Thu Jul 29 22:14:00 2021] rule MonoallelicExpression_pipeline_QC_deseq_qc_R: input: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/allelic_counts/qc_RNA010559.csv.gz, Scripts/MonoallelicExpression/pipeline/QC/deseq_qc.R output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/RNA_GT/RNA010559.Rds log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/qc_RNA010559.Rds jobid: 12 wildcards: rna=RNA010559

Started with deseq Error in DESeqDataSet(se, design = design, ignoreRank) : counts matrix should be numeric, currently it has mode: logical Calls: DESeq4MAE ... deseq_for_allele_specific_expression -> DESeqDataSetFromMatrix -> DESeqDataSet Execution halted [Thu Jul 29 22:14:12 2021] Error in rule MonoallelicExpression_pipeline_QC_deseq_qc_R: jobid: 12 output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_data/mae/RNA_GT/RNA010559.Rds log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/qc_RNA010559.Rds (check log file(s) for error message)

RuleException: CalledProcessError in line 321 of /tmp/tmp_sxafshg: Command 'set -euo pipefail; Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmp9ko8y51q.deseq_qc.R' returned non-zero exit status 1. File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init.py", line 2326, in run_wrapper File "/tmp/tmp_sxafshg", line 321, in rule_MonoallelicExpression_pipeline_QC_deseq_qc_R File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init__.py", line 568, in _callback File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init.py", line 554, in cached_or_run File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init.py", line 2357, in run_wrapper Error in DESeqDataSet(se, design = design, ignoreRank) : counts matrix should be numeric, currently it has mode: logical Calls: DESeq4MAE ... deseq_for_allele_specific_expression -> DESeqDataSetFromMatrix -> DESeqDataSet Execution halted [Thu Jul 29 22:14:13 2021] Error in rule MonoallelicExpression_pipeline_MAE_deseq_mae_R: jobid: 3 output: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/20210706_drop_RBM10/Output/processed_results/mae/samples/D1913781--RNA010559_res.Rds log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.drop/tmp/MAE/deseq/D1913781--RNA010559.Rds (check log file(s) for error message)

RuleException: CalledProcessError in line 284 of /tmp/tmp_sxafshg: Command 'set -euo pipefail; Rscript --vanilla /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/scripts/tmpiwpz50ql.deseq_mae.R' returned non-zero exit status 1. File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init.py", line 2326, in run_wrapper File "/tmp/tmp_sxafshg", line 284, in rule_MonoallelicExpression_pipeline_MAE_deseq_mae_R File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init__.py", line 568, in _callback File "/apps/gent/CO7/skylake-ib/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init.py", line 554, in cached_or_run File "/apps/gent/CO7/skylake-ib/software/snakemake/6.1.0-foss-2020b/lib/python3.8/site-packages/snakemake/executors/init.py", line 2357, in run_wrapper Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /kyukon/data/gent/vo/001/gvo00101/vsc42523/DROP/.snakemake/log/2021-07-29T221349.822586.snakemake.log

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gagneurlab/drop/issues/235#issuecomment-889669434, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABWP745ZVTWJDA2P6SF4T2LT2JDBPANCNFSM5BAOSZ6A .

vyepez88 commented 2 years ago

Hi @svergult , did you have a look at this?

svergult commented 2 years ago

Hi @vyepez88 we indeed had a look at this and got it working.