Closed antoinecarme closed 1 year ago
Large Horizon => Long Term ???
Added movies of prediction intervals for Ozone and airline passengers models for increasing horizons (H=12, 24, 36, 48).
https://github.com/antoinecarme/PyAF_Benchmarks/tree/master/model_visualizer
Add a voting system. Condorcet method by default
https://en.wikipedia.org/wiki/Condorcet_method
Each model has performance measures for each horizon. The value of the performance measure is considered as a horizon vote (H voters). Longer horizons are weighted (the longer the model performs, the better it is).
For the Condorcet method, each pair of models is compared (MAPE values).
In a pair competition , the winning model (smaller MAPE) is assigned a score of 1, this score is reweighted by the horizon length.
$$ Score(M) = \sumh \sum{m != M} h * \mathbb{1}_{MAPE(M , h) < MAPE(m, h)} $$
where $MAPE(m, h)$ is the h-point ahead MAPE for a model m on the validation dataset (multi-step error).
By design, the higher the voting score, the better it is.
H = 10
generated with this script : https://github.com/antoinecarme/pyaf/blob/issue_213_Large_Horizon_Models/tests/long_term_forecasts/test_yosemite_temps_Horizon_10.py
H = 50
H = 100
H = 200
H = 500
H = 800
H = 1000
TODO: Add some documentation about the new model selection procedure, with a detailed example.
For the moment : https://github.com/antoinecarme/pyaf/issues/213#issuecomment-1444231168
TODO: Keep some kind of backward compatibility. Use a new training option to choose between Condorcet and the old method.
Done.
https://github.com/antoinecarme/pyaf/commit/5e496ad8da0c604e4a1896a9a7c06cd789683ff5
TODO : Do some "homework" about existing state of the art methods (FPP3 ?) !!!
Advanced Review Open Access Review of automated time series forecasting pipelines Stefan Meisenbacher, Marian Turowski, Kaleb Phipps, Martin Rätz, Dirk Müller, Veit Hagenmeyer, Ralf Mikut First published: 09 August 2022 https://doi.org/10.1002/widm.1475
https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1475
Fildes, Robert & Petropoulos, Fotios, 2015. "Simple versus complex selection rules for forecasting many time series," Journal of Business Research, Elsevier, vol. 68(8), pages 1692-1701.
https://www.sciencedirect.com/science/article/abs/pii/S0148296315001423
Measuring forecast accuracy Rob J Hyndman Monday, 31 March 2014
Closing.
Large Horizon Models (H large enough). Profiling for CPU/memory/speed.
Compute Prediction intervals for all tested models. Use more sophistical forecast perf combination in model selection (mean ? max ?). decreasing time based weights ? Take into account the shape of the prediction interval (esthetic for model precision).