search

ellisp / forecastHybrid

Convenient functions for ensemble forecasts in R combining approaches from the {forecast} package
GNU General Public License v3.0
80 stars 24 forks source link

Enhanced prediction intervals #19

Closed dashaub closed 6 years ago

dashaub commented 8 years ago

Very conservative prediction intervals are not working (i.e. taking the highest prediction interval value from an individual model for the upper limit and the lowest individual prediction intervals value for the lower limit). We should consider how to enhance this with additional methods that we could expose through a pi.method argument. This could include taking the mean of the model intervals, applying the weights from the models to the intervals, taking the tightest intervals, or bootstrap simulations. There are lots of issues that could arise from the various combination methods (e.g. upper/lower limits that are lower/higher than the mean forecast, etc), so this will need careful thought.

ellisp commented 8 years ago

In what way are they not working (ie are there cases where they are wildly large, and we can see from subsequent reality that this isn't justified)? Given the usual ways of doing prediction intervals for individual models tend to overstate the confidence in the coverage, conservative is probably good.

dashaub commented 8 years ago

This is old, so I can't remember my original point on this. The conservative method of taking the extreme intervals seems to work fine. I'm sure we could put together a more sophisticated bootstrap sampling methodology, but the question becomes if it is computationally justifiable: these ensembles could be very slow already with some of the planned additions of cross validation.

dashaub commented 8 years ago

issue #37 shows a type of problem

ellisp commented 8 years ago

In Rob Hyndman's blog about forecastHybrid and opera he points out the forecastHybrid prediction interval looks far too conservative with his example of monthly CO2 emission data. My feel after all is that high regularity, high frequency data, this method will be too conservative. But we shouldn't rush to change it.

dashaub commented 8 years ago

Yup. I had an email correspondence with him about this issue. To build these intervals correctly, we'd need the variances of each component model error and the covariance matrix for the models. Ideally this could be estimated for each 1 through n step forecast horizon. Then we could derive the SE for the prediction intervals of the ensemble model. Using the in-sample errors/fitted values for this probably isn't a best approach, but now that cvts is done, we could use that. The downside is that cvts would be very slow. Rob told me the "forecast" package recently updated with an hfitted argument that could be used to generate the n-step forecasts that could be used for generating these fitted/residuals as well and would be much faster than the cvts approach, but I haven't had time to try it out yet.

This issue has been known for a while, and I'd like to see it fixed, but parallel is probably my top priority now in terms of missing features. If you're wanting to look into this that would be great. Once we get parallel and the prediction intervals fixed, I'd say that main features are built out in the package.

ganesh-krishnan commented 7 years ago

What is the expected distribution of the prediction intervals for hybrid models. Doesn't appear like it is normal. When we say 80% PI, what distribution does the 80% quantile belong to?

dashaub commented 6 years ago

Added a new argument into forecast.hybridModel(). Default behavior is to use the extreme values from the component forecasts, but a new option "mean" will take the simple average of all the intervals to produce less conservative intervals.

There are obviously superior and more involved solutions, but I'm closing this for now.