maho3 / ltu-ili

Robust ML in Astro
https://ltu-ili.readthedocs.io/en/latest/
45 stars 9 forks source link

Combine TARP and RankStatistics metrics #56

Closed maho3 closed 1 year ago

maho3 commented 1 year ago

Both the TARP and the PlotRankStatistics metrics require you to sample posteriors for all members of the test set. They are currently different classes, requiring you to resample the whole test set twice to calculate both. They should be merged, and the TARP metric should be calculated from the samples already produced in PlotRankStatistics.

Also, add functionality to choose which rank statistics to plot. Maybe its prudent to change the joint metric to 'CheckPosteriorCoverage' or something.

maho3 commented 1 year ago

Note, this will need to account for the pydelfi backend not supporting tarp (Issue #48)

CompiledAtBirth commented 1 year ago

Additional class :PosteriorCoverage(_SampleBasedMetric)

CompiledAtBirth commented 1 year ago

(commit: 68cd48e6d8b11e0ca2414b542a2f0a449329e7cd)