search

peterk87 / nf-villumina

Generic viral Illumina sequence analysis pipeline
MIT License
4 stars 5 forks source link

Fastp process exceeding running time limit #23

Open mnebroski opened 3 years ago

mnebroski commented 3 years ago

I have been trying to run the pipeline on a very large dataset (file sizes >25GB each), and I'm running into an error on the fastp step where it says: Process exceeds running time limit (1h) In the pipeline summary it says the default max time for the workflow is set to 10d, and I've also tried to play around with different times using the --max_time parameter, but it always fails after 1 hour no matter what the max time is set at. Is there another parameter I should be using to adjust this or a way to increase this time another way? Fastp and the subsequent Kraken2 and Centrifuge processes work on a sample that is ~25GB, but the other samples are much larger (>45GB) and it's on those ones that there seems to be the issue.

peterk87 commented 3 years ago

You could try a custom config file with 

process {
  time = 240.h
}

To set the default time limit for all processes to 10 days, for example.

Run pipeline with config file

nextflow run ... -resume -c custom.config ...
mnebroski commented 3 years ago

Thanks I'll try that! If I already have -c set with the villumina-high-mem-centrifuge.config file, can I just add that process for the time to that file so that it will do both? Edit: I tried adding that process to the villumina-high-mem-centrifuge.config file and it seems to be working now! Thanks again!

peterk87 commented 3 years ago

Yes, I believe you can specify multiple configs with the same pipeline analysis.

On Tue, Jul 13, 2021, 09:36 mnebroski @.***> wrote:

Thanks I'll try that! If I already have -c set with the villumina-high-mem-centrifuge.config file, can I just add that process for the time to that file so that it will do both?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/peterk87/nf-villumina/issues/23#issuecomment-879144556, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABACHICPBNLQIIRCBEWZFRTTXRFQTANCNFSM5AHCICPQ .