CDCgov / cfa-viral-lineage-model

Apache License 2.0
10 stars 0 forks source link

Added a new metric (energy score) #15

Closed thanasibakis closed 2 months ago

thanasibakis commented 3 months ago

What's new:

thanasibakis commented 2 months ago

I took a stab at what unit tests could look like for this, as @swo mentioned...

It's not perfect, but if we choose to use the L1 norm everywhere, then these expectations have analytic solutions for the case where the forecast distribution is normal. What I did was generate lineage counts per division-day, compute true lineage proportions per division-day, and then created 1000 "forecasts" of this by sampling from a normal centered around the true proportion.

These clearly aren't valid proportions, as they don't sum to 1 and could even be negative, but the L1 norm and energy score computations don't consider these constraints at all and are still valid. I was able to compare the estimates to what the closed-form solutions dictate they should be, and they match up to decent tolerance

thanasibakis commented 2 months ago

Added the hand-derived unit tests for fixed dataframes as discussed! @swo @afmagee42

pytest passes in the current state of things

thanasibakis commented 2 months ago

(Just rebased on the mkdocs commit)