search

CDCgov / forecasttools-py

A Python package for common pre- and post-processing operations done by CFA Predict for short term forecasting, nowcasting, and scenario modeling.
Apache License 2.0
4 stars 0 forks source link

Model Comparison Infrastructure #15

Open AFg6K7h4fhy2 opened 1 month ago

AFg6K7h4fhy2 commented 1 month ago

Take at least two idata objects with the same fit and forecast period & compare.

Support comparison from

damonbayer commented 1 month ago

Two points:

  1. We should think about comparing n forecasts when designing this
  2. Since scoring is an important part of model comparison, and scoring is done in scoringutils in R, we need to think carefully about how to use both ArviZ and scoringutils.
AFg6K7h4fhy2 commented 1 month ago

Can you be more specific with this?

We should think about comparing $N$ forecasts when designing this.

Not sure what is meant here: (?) $N$ models (with one forecast dataframe per model) (?) $N$ jurisdictions per forecast dataframe (one dataframe per model).

For (2) I agree completely.

damonbayer commented 1 month ago

Your original post mentions comparing 2 forecasts. I think whatever approach we take should be extensible to more than 2 forecasts.

AFg6K7h4fhy2 commented 1 month ago

Your original post mentions comparing 2 forecasts. I think whatever approach we take should be extensible to more than 2 forecasts.

Ah, I see. I should have been specific myself! at least two idata objects

AFg6K7h4fhy2 commented 3 weeks ago

Adding this issue to the Backlog given that multiple, non-trivial issues (e.g. #32 , #18 , #9 , #34 ) remain to be completed before the functionality outlined in this issue will begin to be integrated.