search

AVR-biosecurity-bioinformatics / freyr

A Nextflow-based metabarcoding pipeline for agricultural biosecurity and biosurveillance
0 stars 0 forks source link

Handle extra process time needed for the first pull of a container #25

Open jackscanlan opened 3 weeks ago

jackscanlan commented 3 weeks ago

When using shifter as the container platform, pulling a container for the first time can take much longer than the requested time through SLURM. The check_max + task.attempt code in nextflow.config might help with this in some cases but not always.

Not sure of a good solution. One idea could be to pull all required containers at the start of the pipeline in a dedicated process, but while this might work for shifter, it might not work for other container platforms. It would also require a list of containers to be kept, which isn't a bad idea for software versions etc.

jackscanlan commented 3 weeks ago

Pipeline parameter to flag first-time runs could boost the time for each process by a set amount to help cover container pulling