robbinscalebj
commented
1 year ago Expanding a bit on my previous post:
The overall goal of this project as it was started within the Theory working group (btw, all are welcome!) is to create a variety of automated forecasts that can tell us about the 'realized' predictability of different ecological variables across time and space (others in the Theory working group are looking explicitly at 'intrinsic' predictability). Machine learning models provide a good way to forecast different ecological phenomena in ways that are easily deployable. At some point relatively soon, we want a whole suite to be up and running. As mentioned above, we have an existing forecasting framework set up that can be readily expanded.
If this project is selected, I think some brainstorming followed by coding could be fruitful in accelerating our progress. So, what are the best strategies for improving the utility, interpretability and applicability of the Theory working group models to the goals above, or more broadly? What should be prioritized for coding at Unconference?
-
Model Variety: The 'wishlist' above is all the models currently set up for tidymodels, which we use for existing implementations. Other models might be desirable. Our current thinking is that priority right now may be on improving current forecast structure (below points), rather than increasing the current diversity of algorithms used to fit/predict data. But input would be welcome from the group on what new model types should be prioritized down the line, or even if our Unconference time would be best spent on simply coding and testing new models. This will be an important decision for an Unconference breakout group to make at the start of the meeting and some familiarization with the existing code.
-
Model Structure: Should more drivers/covariates be included than just weather forecasts?
- For instance, a day-of-year variable could be included, or even recent observations of the variable of interest.
-
Uncertainty Propagation: What are good frameworks for propagating uncertainties from these models to make appropriately confident forecasts?
-
Current forecasts don’t account for many sources of uncertainty in the predictions
-
Two potential possibilities: adding a process error from the model residuals or creating conformal prediction intervals using the training data. I am currently coding an approach that estimates parameter uncertainty and calculates process error from the residuals, but input would be exceedingly welcome. We may run into computational challenges (allowable run time with Git Actions) if the ensembles become too large, but I don’t think that’s a problem yet as these models fit very quickly.
-
Ancillary Data/Documentation: What might impede future expansion, use, or interpretation of these forecasts that could be remedied now?
- Useful convention for renaming different flavors of existing forecasts?
- Useful variables to track on data quality, model fits, 'surprises', etc., that could inform forecast interpretation when the forecasts are finally analyzed together.
-
The Theory working group has been automating forecasts in a machine learning framework. Forecasts fit with linear regression, LASSO regression, random forest, and bagged multi-layer perceptron (a neural network) are actively submitting to the challenge. We developed the code as somewhat of a plug 'n' play for tidymodels, with code here: https://github.com/eco4cast/Forecast_submissions/tree/main/Generate_forecasts.
Some needs include:
More model types - There's a 'wishlist' for models that already exist as tidymodels engines: https://docs.google.com/spreadsheets/d/1SjcwRDXQI2GmmJwgy7K91-RP_wMp34x37SiDJm1AbI8/edit#gid=0
Better representation of uncertainty, which right now is only from drivers/predictor ensembles (i.e., NOAA weather).
Additional predictors or predictor sets - currently, just NOAA weather forecasts as model drivers.
Improving model and training documentation. Currently, hyperparameters from training, training rmse, and number of training observations are documented, but other information could be interesting for future use but haven't been included or thought of.