Roadmap (2019 July)

[ha0ye]

Objective:

• measure complexity (weighted permutation entropy, others?) and forecast skill across our collection of time series
• identify what properties affect forecast skill; how these properties can guide selection of forecast models

Procedure:

• restrict to 1-step ahead forecasts (note that this can have different time steps, depending on sampling frequency)
• begin with 2/3 training, 1/3 testing split (as in Pennekamp et al. 2019). We can also look at 3-fold cross-validation (to check against long-term changes in dynamics)
• models:
  • [x] linear autoregressive
  • [x] exponential autoregressive
  • [x] nonlinear time-delay embedding
  • [x] random walk / state-space
  • [ ] linear/quadratic/splined functions of time

Datasets:

• existing real datasets in MATSSforecasting and MATSS
  • check for duplications of time series
• simulations of certain broad classes of population dynamics

Predictors:

• lifespan (and life history characteristics)
• temperature (min / max / mean of center of range or geo coordinates)
• time series length, sampling freq, autocorrelation, perm. entropy

[ha0ye]

After digging into the built-in arima() function and autoarima() and arfima() (from the forecast package, designed to operate similarly), the procedure is a bit more constrained.

These functions are designed to forecast forward in time only (and not as generic models for {state at time t} -> {state at time t+1}). So trying to re-use a fitted model and apply it to different portions of the time series is not really doable. (I suspect this is because of how Kalman filter and similar are operating under the hood that estimate true state vs. observations.)

So instead of trying to do cross-validation or use a static model for multiple forecasts, I'm moving over to a hindcasting approach:

1. fit the model to the first 2/3 of the time series
2. make one prediction
3. add the next observation to the set of data for fitting, and go back to 1.

We can use a parameter for when to start hindcasting. This will support later tests of how much data is needed, relative to model selection, etc.