edwardbirdlab / BALROG-MON

A Nextflow pipeline for Antimicrobial Resistatnce exploration in metagenomic samples
Other
2 stars 1 forks source link

Improvements for USDA ARS systems. #9

Closed molikd closed 5 months ago

molikd commented 5 months ago

A slight delay should be added before jobs ends, perhaps adjustable, as a ARS HPCs tend to have network congestion around disk writing. So cfg process should have delays in the config.

something akin to

  errorStrategy { sleep(Math.pow(2, task.attempt) * 200 as long); return 'retry' }
  maxRetries 5

see: https://www.nextflow.io/docs/latest/process.html#dynamic-retry-with-backoff