Closed thanasibakis closed 2 months ago
I took a stab at what unit tests could look like for this, as @swo mentioned...
It's not perfect, but if we choose to use the L1 norm everywhere, then these expectations have analytic solutions for the case where the forecast distribution is normal. What I did was generate lineage counts per division-day, compute true lineage proportions per division-day, and then created 1000 "forecasts" of this by sampling from a normal centered around the true proportion.
These clearly aren't valid proportions, as they don't sum to 1 and could even be negative, but the L1 norm and energy score computations don't consider these constraints at all and are still valid. I was able to compare the estimates to what the closed-form solutions dictate they should be, and they match up to decent tolerance
Added the hand-derived unit tests for fixed dataframes as discussed! @swo @afmagee42
pytest
passes in the current state of things
(Just rebased on the mkdocs commit)
What's new:
proportions_mae
is nowproportions_mean_norm
, to be a bit more correct in our naming