Open fssem1 opened 3 years ago
Conclusions from June 15, 2021 Preseason Forecast Meeting: • Bootstrap CI versus prediction interval on the forecast? o The decision was made to go with the prediction interval. • Use model averaging (what models to include) or just go with the top model based on a performance metric as AICc, MASE etc. and not use model averaging? o The decision was made to go with model averaging but to weight the models based on a different metric than AIC (e.g., MAPE_LOOCV, MAPE_one_step_ahead, wMAPE but weight the historical years in a decreasing fashion) o Additional references (thanks to Rich): https://www.cs.cmu.edu/~schneide/tut5/node42.html https://www.adfg.alaska.gov/static/applications/dcfnewsrelease/1232415165.pdf https://www.stat.umn.edu/geyer/5421/slides/glmbb.html https://uoftcoders.github.io/rcourse/lec09-model-selection.html https://cran.r-project.org/web/packages/AICcmodavg/vignettes/AICcmodavg.pdf https://rdrr.io/cran/MuMIn/man/model.avg.html https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1309 https://theoreticalecology.wordpress.com/2018/05/14/model-averaging-in-ecology-a-review-of-bayesian-information-theoretic-and-tactical-approaches-for-predictive-inference/ o Rich and Sara will see if they can find any support for one performance metric over another. • Move to just the 20 m ISTI variable for the SECM variables for the 2022 forecast? o The decision was made to just stick with ISTI_20m_MJJ variable as a possible variable in the models; The other ISTI variables will not be included as possible variables in the model averaging process for the 2022 preseason forecast. • Logical set of environmental variables (space/time) to assess?—25 models currently in the document o Reduced set of variables that includes:
Conclusions from June 15, 2021 Preseason Forecast Meeting: • Bootstrap CI versus prediction interval on the forecast? o The decision was made to go with the prediction interval. • Use model averaging (what models to include) or just go with the top model based on a performance metric as AICc, MASE etc. and not use model averaging? o The decision was made to go with model averaging but to weight the models based on a different metric than AIC (e.g., MAPE_LOOCV, MAPE_one_step_ahead, wMAPE but weight the historical years in a decreasing fashion) o Additional references (thanks to Rich): https://www.cs.cmu.edu/~schneide/tut5/node42.html https://www.adfg.alaska.gov/static/applications/dcfnewsrelease/1232415165.pdf https://www.stat.umn.edu/geyer/5421/slides/glmbb.html https://uoftcoders.github.io/rcourse/lec09-model-selection.html https://cran.r-project.org/web/packages/AICcmodavg/vignettes/AICcmodavg.pdf https://rdrr.io/cran/MuMIn/man/model.avg.html https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1309 https://theoreticalecology.wordpress.com/2018/05/14/model-averaging-in-ecology-a-review-of-bayesian-information-theoretic-and-tactical-approaches-for-predictive-inference/ o Rich and Sara will see if they can find any support for one performance metric over another. • Move to just the 20 m ISTI variable for the SECM variables for the 2022 forecast? o The decision was made to just stick with ISTI_20m_MJJ variable as a possible variable in the models; The other ISTI variables will not be included as possible variables in the model averaging process for the 2022 preseason forecast. • Logical set of environmental variables (space/time) to assess?—25 models currently in the document o Reduced set of variables that includes: