populationgenomics / automated-interpretation-pipeline

Rare Disease variant prioritisation MVP

MIT License

5 stars 4 forks source link

Fix AIP as pipeline stages #325

Closed MattWellie closed 6 months ago

MattWellie commented 6 months ago

Fixes

Relates to https://github.com/populationgenomics/production-pipelines/issues/489
AIP runs on one dataset, which doesn't translate to easy pipeline running
An error created during the linter changes meant that remote JSON files weren't being read properly

Proposed Changes

Undoes the Path/cpg_utils.Path type overloading (causing JSON parse failure)
Each AIP module required in the pipeline takes an optional --dataset argument, so that we can parallelise without breaking everything
A couple of config loading issues have been corrected

Test run-thru https://batch.hail.populationgenomics.org.au/batches/429203

Checklist

[x] Related Issue created
[x] Tests covering new change
[x] Linting checks pass

This has been tested in -test on the validation dataset (all except the final stage which was tested locally)

Twinned with https://github.com/populationgenomics/production-pipelines/pull/490