search

dssg / triage

General Purpose Risk Modeling and Prediction Toolkit for Policy and Social Good Problems
Other
185 stars 60 forks source link

Audition needs to handle many models per as_of_date in a model group #800

Closed nanounanue closed 3 years ago

nanounanue commented 4 years ago

If several models with different random seed per as of date are inside a model group, is important to check the stability of the metric regarding the fluctuations caused by the random seed.

One easy way to solve this, is ignoring and just get the avg or another aggregation in the query before using audition.