search

CDCgov / forecasttools-py

A Python package for common pre- and post-processing operations done by CFA Predict for short term forecasting, nowcasting, and scenario modeling.
Apache License 2.0
1 stars 0 forks source link

Model Comparison Infrastructure #15

Open AFg6K7h4fhy2 opened 3 days ago

AFg6K7h4fhy2 commented 3 days ago

Take at least two idata objects with the same fit and forecast period & compare.

Support comparison from

damonbayer commented 3 days ago

Two points:

  1. We should think about comparing n forecasts when designing this
  2. Since scoring is an important part of model comparison, and scoring is done in scoringutils in R, we need to think carefully about how to use both ArviZ and scoringutils.
AFg6K7h4fhy2 commented 3 days ago

Can you be more specific with this?

We should think about comparing $N$ forecasts when designing this.

Not sure what is meant here: (?) $N$ models (with one forecast dataframe per model) (?) $N$ jurisdictions per forecast dataframe (one dataframe per model).

For (2) I agree completely.

damonbayer commented 3 days ago

Your original post mentions comparing 2 forecasts. I think whatever approach we take should be extensible to more than 2 forecasts.

AFg6K7h4fhy2 commented 3 days ago

Your original post mentions comparing 2 forecasts. I think whatever approach we take should be extensible to more than 2 forecasts.

Ah, I see. I should have been specific myself! at least two idata objects