Prediction Interval Questions

[webert6]

Hello, I just want to make sure I am understanding correctly about how prediction intervals are used. I've been reading the documentation and code, but it's a bit difficult to comprehend without pulling apart the code executing it line by line.

• If a list of levels is specified, the forecast method uses the native method for prediction intervals, correct?

• To get conformal prediction intervals, I must specify that within the model parameters using statsforecast.utils.ConformalIntervals? But if the model method, like ADIDA, doesn't have native prediction intervals, does it automatically use conformal prediction intervals?

• Are the conformal prediction intervals constant across each step of a future horizon?

Your feedback is much appreciated.

[jmoralez]

Hey @webert6, thanks for using statsforecast.

• Yes.
• If a model doesn't natively support prediction intervals and you don't provide a configuration for conformal you'll get an error. In those cases you can specify them within each model or also in the StatsForecast methods like fit, forecast, etc. and they get applied to all models.
• The scores are computed by serie and horizon, so if you have 1 serie and use 4 windows with a horizon of 10 you'll have 40 scores.

Please let us know if you have further doubts.

[webert6]

Thank you for the quick reply @jmoralez. I just have a couple more questions as I'm still wrapping my head around conformal prediction.

• Are the conformal prediction intervals calculate differently if a forecast method has a probabilistic component?

• Do you have any guidelines for selecting the proper parameters for conformal prediction?

[jmoralez]

• If you set the interval settings for a model that natively supports intervals it will override them and provide the conformal ones.
• The h must match your horizon and more windows provide a better estimate, 4 is probably fine but depending on the length of your series and your time budget you could use more or less.

[webert6]

Regarding the first question, if the conformal prediction intervals differ if the specified model has a probabilistic output natively, I was watching this talk on conformal prediction and she mentioned combining probabilistic forecasting models with conformal prediction intervals can provide better prediction intervals than simply using conformal.

https://youtu.be/--WcrDRtrYk?si=Xc-cQRbgJ0ohoigN

Again, thank you for your help understanding.

[jmoralez]

If you want to you can define the same model twice with different settings and combine its intervals (the mean should be the same), e.g.

from statsforecast import StatsForecast
from statsforecast.models import AutoARIMA
from statsforecast.utils import AirPassengersDF, ConformalIntervals

sf = StatsForecast(
    models=[
        AutoARIMA(season_length=12, alias='native'),
        AutoARIMA(
            season_length=12,
            prediction_intervals=ConformalIntervals(n_windows=4, h=12),
            alias='conformal',
        ),
    ],
    freq='M',
)
sf.forecast(df=AirPassengersDF, h=12, level=[80])

[webert6]

Great thanks! Really appreciate the responsiveness