fssem1
commented
3 years ago Based on Curry’s response…
-
I think 1-step ahead forecasts should be used with MAPE or MASE (instead of LOOCV with MAPE or MASE).
-
If we go with model averaging (which Curry is calling multi-model inference), he is saying to weight the models by the inverse-variance where the variance is calculated as observed and predicted abundance (or biomass) across time in log space. This would be instead of weighting the models by AIC.
I think we could forward the response to the group with the two conclusions above and see if there is agreement to go forward.
rich-brenner
jmurphygithub
From: Curry Cunningham cjcunningham@alaska.edu Sent: Thursday, June 24, 2021 8:03 PM To: Brenner, Richard E (DFG) richard.brenner@alaska.gov Subject: Re: SECM Pink Salmon Forecast Meeting
Hi Rich,
Hope you are doing well, thanks for reaching out.
I have some personal preferences in some of these areas.
I like evaluating 1-step ahead predictive performance, as it makes intuitive given the way we collect data and generate salmon forecasts. In addition 1-step ahead performance should provide a more fair assessment if you have (a) unmodeled autocorrelation in your system, or (b) long term trends or shifts in the average expected value (i.e. regime shifts).
Leave-one-out cross-validation, while often used on the ML end, doesn’t make as much sense with respect to the type of small and likely autocorrelated datasets we have for salmon abundance… at least in my opinion.
I’m always keen to ask “Would we have done better or worse in year X, with this alternative model, given the data we had in hand at that time (likely through year X-1).
For multi-model inference, I tend to prefer Inverse-variance of the ensemble members. Where the variance is the realized difference between observed and predicted abundance (or biomass) across time in log space. You can always do it in normal space for simplicity and to avoid nit-picky discussions of lognormal bias correction, but in reality abundance forecast errors tend to be log-normally distributed in my experience.
There are of course useful (and more fruitful) Bayesian approaches to model weighting, but they generally will do the same thing… Weight each prediction in proportion to its past predictive performance.
runSize ~ prior
Likelihoods
log(pred_1) ~ Normal(runSize, sigma_pred_1) log(pred_2) ~ Normal(runSize, sigma_pred_2)
Where sigma_pred_1 and sigma_pred_2 are the lognormal standard deviations, representing past model performances across a comparable time series.
In the Stan context this looks like the following for my Bristol Bay (and our forthcoming Yukon) model
target += normal_lpdf(log(pred_ce+1e-6) | log(pred+1e-6), sigma_ce);
Where “pred” is equivalent to runSize above.
With respect to performance metrics, I usually tend to like: • MAPE - It is interpretable and easy to explain to stakeholders. Most folks think in % • RMSE - Not bounded by asymmetry associated with zero-bounded values. • MASE - Mean absolute scaled error • Good old Rsq
My Bristol Bay GCI cell phone is: 500-8323, happy to chat anytime.
Cheers,
Curry
On Jun 24, 2021, at 2:48 PM, Brenner, Richard E (DFG) richard.brenner@alaska.gov wrote:
Greetings Curry~
Can you provide any perspective on the use of mean absolute percent error with either ‘leave-one-out with cross-validation’ or ‘one-step-ahead’ approaches for selecting a best model for the Southeast pink salmon forecast? We’re at a bit of an impasse!
I hope that all is well there!
Rich
From: Miller, Sara E (DFG) sara.miller@alaska.gov Sent: Monday, June 7, 2021 1:32 PM To: Piston, Andrew W (DFG) andrew.piston@alaska.gov; Emily Fergusson - NOAA Federal emily.fergusson@noaa.gov; Andrew Gray - NOAA Federal andrew.gray@noaa.gov; Heinl, Steve (DFG) steve.heinl@alaska.gov; Jim Murphy jim.murphy@noaa.gov; Wes Strasburger - NOAA Federal wes.strasburger@noaa.gov; jamal.moss@noaa.gov; Brenner, Richard E (DFG) richard.brenner@alaska.gov Subject: RE: SECM Pink Salmon Forecast Meeting
Hello all- Attached are the updated temperature variables (satellite_SST_process_4_June_2021.pdf) and the modelling forecast process document using these temperature variables (preliminary_discussion_about_2022_forecast_process_4_June_2021.pdf). These will serve as the discussion pieces for next week’s SECM meeting. Some topics that need to be addressed at the meeting include:
• Bootstrap CI versus prediction interval on the forecast; • model averaging (what models to include) or just go with the top model based on a performance metric as AICc, MASE etc. and not use model averaging; • moving to just the 20 m ISTI variable for the SECM variables; and • logical set of environmental variables (space/time) to assess—25 models currently in the document.
Feel free to add more topics to this email chain so that we can cover them all.
Sara Miller
From: Piston, Andrew W (DFG) andrew.piston@alaska.gov Sent: Tuesday, June 1, 2021 2:00 PM To: Emily Fergusson - NOAA Federal emily.fergusson@noaa.gov; Andrew Gray - NOAA Federal andrew.gray@noaa.gov Cc: Heinl, Steve (DFG) steve.heinl@alaska.gov; Jim Murphy jim.murphy@noaa.gov; Wes Strasburger - NOAA Federal wes.strasburger@noaa.gov; jamal.moss@noaa.gov; Miller, Sara E (DFG) sara.miller@alaska.gov; Brenner, Richard E (DFG) richard.brenner@alaska.gov Subject: RE: SECM Pink Salmon Forecast Meeting
Thanks Emily, The summary Sara will send out has a lot of detail in so you can always make some e-mail comments if you have any strong thoughts on how to proceed.
Andy
From: Emily Fergusson - NOAA Federal emily.fergusson@noaa.gov Sent: Tuesday, June 1, 2021 11:46 AM To: Andrew Gray - NOAA Federal andrew.gray@noaa.gov Cc: Piston, Andrew W (DFG) andrew.piston@alaska.gov; Heinl, Steve (DFG) steve.heinl@alaska.gov; Jim Murphy jim.murphy@noaa.gov; Wes Strasburger - NOAA Federal wes.strasburger@noaa.gov; jamal.moss@noaa.gov; Miller, Sara E (DFG) sara.miller@alaska.gov; Brenner, Richard E (DFG) richard.brenner@alaska.gov Subject: Re: SECM Pink Salmon Forecast Meeting
Hi All,
I will be out of town from June 3-21st. You can let Andy G. and I know what grandiose decisions you made in our absence. :)
Emily
On Tue, Jun 1, 2021 at 11:01 AM Andrew Gray - NOAA Federal andrew.gray@noaa.gov wrote: I will be out of town from June 5th to the 23rd. I can get filled in after I get back if need be... Best, Andy
The contents of this message are mine personally and do not necessarily reflect any position of NOAA
Andrew K Gray, Supervisor Salmon Ocean Ecology & Bycatch Analysis Auke Bay Laboratories Alaska Fisheries Science Center, NOAA Fisheries
Ted Stevens Marine Research Institute 17109 Point Lena Loop Rd. Juneau, AK 99801 USA (907)789-6047
On Tue, Jun 1, 2021 at 9:23 AM Piston, Andrew W (DFG) andrew.piston@alaska.gov wrote: Hello Everyone, Sara Miller has recently completed a summary of the myriad of different temperature variables we outlined in our last meeting and we are hoping to set up another teleconference to discuss the results and ways to move forward with the forecast. After an initial preview by Jim, myself and Steve, Sara is adding a couple more items to her summary and should have a new edition our for everyone soon. We were hoping to have a teleconference to go over the summary on June 14th or 15th, with the 15th being preferred at the moment (I will be in the field for a week starting the 16th). Let me know if Tuesday, June 15th at 10:00 a.m. works for you to have a teleconference, or if you another time on the 14th or 15th works better. Once I hear from everyone I will set up a TEAMS meeting.
Thanks,
Andy Piston Alaska Department of Fish and Game SEAK Pink and Chum Salmon Project Leader Pacific Salmon Commission Northern Panel 2030 Sea Level Drive, Suite 205, Ketchikan, AK 99901 907-225-9677
-- Emily Fergusson
Fishery Research Biologist Auke Bay Laboratories Alaska Fisheries Science Center, NOAA Fisheries 17109 Point Lena Loop Road Juneau, AK 99801 TEL 907-789-6613