search

CDCgov / multisignal-epi-inference

Python package for statistical inference and forecast of epi models using multiple signals
https://cdcgov.github.io/multisignal-epi-inference/
9 stars 1 forks source link

Create forecasting interface #169

Open damonbayer opened 2 weeks ago

damonbayer commented 2 weeks ago

Goal

We should have a consistent interface for creating forecasts from models.

Context

For implementation inspiration, we can look at the NumPyro Time Series Forecasting example.

SamuelBrand1 commented 2 weeks ago

Following from f2f discussion, its worth considering this problem: https://github.com/CDCgov/Rt-without-renewal/issues/243

Underlying difficulty is using Predictive, you should check if it works on an AR process like here https://num.pyro.ai/en/latest/examples/ar2.html because its a different inference process (stochastic even when conditioned on parameters).

damonbayer commented 1 week ago

There is some discussion of forecasting for the AR2 NumPyro demo here: https://github.com/pyro-ppl/numpyro/pull/1350

damonbayer commented 1 week ago

See also https://juanitorduz.github.io/arma_numpyro/

SamuelBrand1 commented 1 week ago

See also https://juanitorduz.github.io/arma_numpyro/

This is nice example... but looks a bit weird?

I think statsmodel use MLE inference under the hood? In which case I'm not sure I'd expect the forecast to look so similar to one fitted with NUTS?

damonbayer commented 1 week ago

@SamuelBrand1 weird in what way?

SamuelBrand1 commented 1 week ago

That it looks so similar to statsmodel given its a different inference method? It doesn't look like the posteriors are super peaked?

SamuelBrand1 commented 1 week ago

Looking at the code, the predictive mode switch looks like it makes sense.

damonbayer commented 3 days ago

Does anyone have a forecasting interface (i.e. the function signature for creating a forecast) that they really like that I might use for inspiration? Could be in Python, R, Julia, etc. @SamuelBrand1 @seabbs @dylanhmorris

damonbayer commented 3 days ago

This discussion on the numpyro forum looks promising.

Also noting that there is a GaussianRandomWalk class built in to numpyro that we could take inspiration from.

SamuelBrand1 commented 2 days ago

This discussion on the numpyro forum looks promising.

Yeah, this example points at what I was worried about with forecasting (i.e. when you have a stochastic distribution in the future as opposed to the time series example with numpyro docs).

From my basic looking around it seems that the most common solution in numpyro land is to have some kind of conditional statement inside the def of the sampleable model... that seems to do the job but maybe seems like potentially a lot of boiler plate? E.g. would we get pushed towards needing to write an extra conditional chunk of all the models at the top level that we want to forecast from?

SamuelBrand1 commented 2 days ago

Also noting that there is a GaussianRandomWalk class built in to numpyro that we could take inspiration from.

Is there a downside to just importing it? If its been provided by numpyro devs then it might be better in some way that what we do ourselves?

SamuelBrand1 commented 2 days ago

Does anyone have a forecasting interface (i.e. the function signature for creating a forecast) that they really like that I might use for inspiration? Could be in Python, R, Julia, etc. @SamuelBrand1 @seabbs @dylanhmorris

I like the epidemia interface for prediction/counterfactual: https://imperialcollegelondon.github.io/epidemia/reference/posterior_predict.epimodel.html .

The tl; dr explanation is they fit an object (which contains posterior draws for the params) and then passing newdata can create forecasts (by being longer) and/or counterfactuals (by having different covariates like an earlier lockdown). Its demoed here, https://imperialcollegelondon.github.io/epidemia/articles/europe-covid.html .