populationgenomics / automated-interpretation-pipeline

Rare Disease variant prioritisation MVP

MIT License

5 stars 4 forks source link

Implement backoff retrier for HTTP requests #337

Closed MattWellie closed 6 months ago

MattWellie commented 6 months ago

Fixes

Running multiple AIP runs in parallel (i.e. through the main pipeline) results in a flood of PanelApp queries that kill all the jobs
e.g. https://batch.hail.populationgenomics.org.au/batches/429865/jobs/4

Proposed Changes

implements a backoff retry wrapper to catch failed queries, and re-attempt over increasing intervals
eventually accepts failure, but only after a bunch of retries

Extra

prints the config contents once at the start of the clinvar & main pipeline runs, in case https://github.com/populationgenomics/cpg-utils/pull/146 is accepted