broadinstitute / lrma-aou1-panel-creation

Pipelines and evaluations covering integration, phasing, and imputation of short and structural variants for the AoU Phase 1 long-reads callset.
1 stars 0 forks source link

Attempt to tune Hapestry parameters (maybe with Optuna WDL)? #51

Open rlorigro opened 1 month ago

rlorigro commented 1 month ago

We are starting to accumulate some parameters in hapestry, which could benefit from parameter tuning (as with Optuna).

To start with, these seem like good candidates (in order of priority):

As an objective function for Optuna, the F1 of vcfdist could be used, either directly on the VCF or on the final output of the phasing pipeline.

rlorigro commented 1 month ago

Changed title because just 2 parameters might be simple enough with a regular calibration curve instead of using Optuna. Third param is somewhat optional, if we don't consider run time. Opinions?

samuelklee commented 1 month ago

Hmm, this might be small and interpretable enough that a rough grid search might be warranted instead? There will be some development overhead and a bit of a learning curve required to get the Optuna notebook up and running. But if you want to take a look, I think the Malaria_optuna_test.ipynb notebook in the workspace I just shared with you (https://app.terra.bio/#workspaces/broad-firecloud-dsde/malaria-filtering-optimization-staging_monica%20copy) might be the place to start.

samuelklee commented 1 month ago

And just to be clear, the approach here is to run Optuna within a notebook that launches your WDL with appropriate parameters. So you could even use a similar approach to do the grid search.

rlorigro commented 1 month ago

yea on second thought, I can just make a Terra table with all the parameters as rows and run them. Will probably be easier than using the notebook

rlorigro commented 1 month ago

Made some progress on this, using Fabio's latest addition to Hapestry which includes small vars. More tests will be needed at the 1074 sample scale, but this is where we are at now: image image

all results are computed with vcfdist --realign-etc over the chr1:100Mbp-110Mbp region using 47 hprc samples

rlorigro commented 1 month ago

I've decided to accept the polynomial difference-from-best weighting scheme because it appears to be more stable and reach a higher maximum. Unfortunately Precision and Recall in this context are not very meaningful, so we are just using F1 to guide our decisions.