nf-core / circrna

circRNA quantification, differential expression analysis and miRNA target prediction of RNA-Seq data
https://nf-co.re/circrna
MIT License
44 stars 21 forks source link

Not all samples are used in analysis resulting in pipeline failur #80

Closed npatel-ah closed 8 months ago

npatel-ah commented 11 months ago

Description of the bug

Hello,

Thank you for the pipeline. I am having quite trouble fully running the pipeline. I tried it with both Docker and Singularity with both master and dev branches and every time different issues occurs. I am listing the one is key for me to move forward,

In the dev branch

I have 6 samples but after the TRIMGALORE/FASTQ steps, it only runs one sample with STAR 1st pass. After failing with I try to resume the pipeline, somehow the STAR part shows compeletly different sample.

Command used and terminal output

Command

nextflow run nf-core/circrna  \
--input 'samplesheet.csv' \
--phenotype 'phenotype.csv' \
--genome 'GRCm38' \
--outdir 'Analysis' \
--tool 'circexplorer2' \
--module 'circrna_discovery,mirna_prediction,differential_expression' \
--mature '/BI/GenomeDB/Mus_musculus/GRCm38.102/mature_mus_musculus.fa' \
--fasta '/BI/GenomeDB/Mus_musculus/GRCm38.102/Mus_musculus.GRCm38.102.fa' \
--gtf '/BI/GenomeDB/Mus_musculus/GRCm38.102/Mus_musculus.GRCm38.102.gtf' \
--hisat2 '/BI/GenomeDB/Mus_musculus/GRCm38.102/hisat2_cricRNA' \
--star '/BI/GenomeDB/Mus_musculus/GRCm38.102/STAR' \
--igenomes_base /BI/GenomeDB \
--species Grcm38 -resume -profile docker --bsj_reads 2 -r dev

Try1

[-        ] process > NFCORE_CIRCRNA:CIRCRNA:PREPARE_GENOME:STAR_GENOMEGENERATE                                                   -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:PREPARE_GENOME:SEGEMEHL_INDEX                                                        -
[c7/37137c] process > NFCORE_CIRCRNA:CIRCRNA:FASTQC_TRIMGALORE:FASTQC (Sample-04)                                               [100%] 6 of 6, cached: 6 ✔
[bf/5da299] process > NFCORE_CIRCRNA:CIRCRNA:FASTQC_TRIMGALORE:TRIMGALORE (Sample-04)                                           [100%] 6 of 6, cached: 6 ✔
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SEGEMEHL_ALIGN                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SEGEMEHL_FILTER                                                    -
[d3/51cbba] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:STAR_1ST_PASS (Sample-03)                                        [100%] 1 of 1 ✔
[38/8a4c88] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:STAR_SJDB (star_sjdb)                                              [100%] 1 of 1 ✔
[7f/b54d3b] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:STAR_2ND_PASS (Sample-03)                                        [100%] 1 of 1 ✔
[02/6c4629] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_REF (Mus_musculus.GRCm38.102.gtf)                    [100%] 1 of 1, cached: 1 ✔
[2d/2cef78] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_PAR (Sample-03)                                    [100%] 1 of 1 ✔
[9a/848547] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_ANN (Sample-03)                                    [100%] 1 of 1 ✔
[85/958c3b] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_FLT (Sample-03)                                    [100%] 1 of 1 ✔
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCRNA_FINDER_FILTER                                              -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC_ALIGN                                                    -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SAMTOOLS_INDEX                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SAMTOOLS_VIEW                                                      -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC_ANCHORS                                                  -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC                                                          -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC_FILTER                                                   -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRIQUANT_YML                                                      -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRIQUANT                                                          -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRIQUANT_FILTER                                                   -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_1ST_PASS                                                       -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_SJDB                                                           -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_2ND_PASS                                                       -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_MATE1_1ST_PASS                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_MATE1_SJDB                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_MATE1_2ND_PASS                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_MATE2_1ST_PASS                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_MATE2_SJDB                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_MATE2_2ND_PASS                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC                                                                -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:DCC_FILTER                                                         -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:MAPSPLICE_REFERENCE                                                -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:MAPSPLICE_ALIGN                                                    -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:MAPSPLICE_PARSE                                                    -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:MAPSPLICE_ANNOTATE                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:MAPSPLICE_FILTER                                                   -
[22/84f47e] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:ANNOTATION (Sample-03:circexplorer2)                             [  0%] 0 of 1
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FASTA                                                              -
[be/f37df8] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:COUNTS_SINGLE (circexplorer2)                                      [100%] 1 of 1 ✔
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:MIRNA_PREDICTION:TARGETSCAN_DATABASE                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:MIRNA_PREDICTION:TARGETSCAN                                                          -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:MIRNA_PREDICTION:MIRANDA                                                             -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:MIRNA_PREDICTION:MIRNA_TARGETS                                                       -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:HISAT2_ALIGN                                                 -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT                        -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_INDEX                       -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_STATS    -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_FLAGSTAT -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:BAM_SORT_STATS_SAMTOOLS:BAM_STATS_SAMTOOLS:SAMTOOLS_IDXSTATS -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:STRINGTIE_STRINGTIE                                          -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:STRINGTIE_PREPDE                                             -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:DESEQ2_DIFFERENTIAL_EXPRESSION                               -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:PARENT_GENE                                                  -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:PREPARE_CLR_TEST                                             -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:DIFFERENTIAL_EXPRESSION:CIRCTEST                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CUSTOM_DUMPSOFTWAREVERSIONS                                                          -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:MULTIQC                                                                              -
-[nf-core/circrna] Pipeline completed with errors-
WARN: Killing running tasks (1)

Try2
[17/7a55ab] process > NFCORE_CIRCRNA:CIRCRNA:PREPARE_GENOME:HISAT2_BUILD (Mus_musculus.GRCm38.102.fa)                             [  0%] 0 of 1
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:PREPARE_GENOME:STAR_GENOMEGENERATE                                                   -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:PREPARE_GENOME:SEGEMEHL_INDEX                                                        -
[a1/cea211] process > NFCORE_CIRCRNA:CIRCRNA:FASTQC_TRIMGALORE:FASTQC (Sample-06)                                               [100%] 6 of 6, cached: 6 ✔
[68/c1e594] process > NFCORE_CIRCRNA:CIRCRNA:FASTQC_TRIMGALORE:TRIMGALORE (Sample-03)                                           [100%] 6 of 6, cached: 6 ✔
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SEGEMEHL_ALIGN                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SEGEMEHL_FILTER                                                    -
[49/33bd8f] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:STAR_1ST_PASS (Sample-02)                                        [100%] 1 of 1, cached: 1 ✔
[45/93808b] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:STAR_SJDB (star_sjdb)                                              [100%] 1 of 1, cached: 1 ✔
[e5/fd8186] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:STAR_2ND_PASS (Sample-02)                                        [100%] 1 of 1, cached: 1 ✔
[76/4c2152] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_REF (Mus_musculus.GRCm38.102.gtf)                    [100%] 1 of 1, cached: 1 ✔
[08/a8caeb] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_PAR (Sample-02)                                    [100%] 1 of 1, cached: 1 ✔
[76/4ca3bc] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_ANN (Sample-02)                                    [100%] 1 of 1, cached: 1 ✔
[52/c81cf8] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_FLT (Sample-02)                                    [100%] 1 of 1, cached: 1 ✔
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCRNA_FINDER_FILTER                                              -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC_ALIGN                                                    -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SAMTOOLS_INDEX                                                     -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:SAMTOOLS_VIEW                                                      -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC_ANCHORS                                                  -
[-        ] process > NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:FIND_CIRC                                                          -

Relevant files

No response

System information

nextflow version 23.04.1 Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-213-generic x86_64) Container Docker Hardware Local Linux

nf-core/circrna -r dev

rreggiar commented 11 months ago

I've been so focused on fixing other obvious breakpoints in my use case of this that I just confirmed I am also seeing this occur with the Singularity + Slurm. I have an unrelated issue + PR coming up but I'm going to take a look at this too, not sure I can help but will try since I've been working with this for a bit now.

npatel-ah commented 11 months ago

Thank you much for quick response. I will keep an eye on updates.

rreggiar commented 11 months ago

Seems like this works as intended with test_full

In my production runs I'm also providing explicit paths to local references and indexes....I wonder if that could be causing something

rreggiar commented 11 months ago

I don't have time quite yet for a PR but fixing this is caused by multiple processes trying to access star_index while it isn't being collected in the pipeline. Replacing all process arg instances of start_index with star_index.collect() fixes

nictru commented 10 months ago

Hey everyone, sorry for the late response I think I already fixed this, will have my branch merged ASAP and then we can have a second look

rreggiar commented 10 months ago

Yeah, I've done quite a bit of re-work on this pipeline since I found a lot of issues with channel mix-up preventing completion in addition to this... my fork is going to take a bit to merge into dev though since I prioritized functionality on my compute environment over generalizability. @nictru have you also seen stuff like this? It needs a separate issue and I'll try and get a PR up soon but figure I'd check in here.

nictru commented 10 months ago

I also made some larger changes which are currently on the single-stranded branch (the name might be a bit misleading but it was the first reason to create the branch)

To make communication a bit easier, you could text me in the nf-core slack - just head to the nf-core/circrna channel, you will find me