search

antoinecarme / pyaf

PyAF is an Open Source Python library for Automatic Time Series Forecasting built on top of popular pydata modules.
BSD 3-Clause "New" or "Revised" License
459 stars 73 forks source link

Evaluate Continuous Ranked Probability Score as a Forecasting Performance Measure #74

Closed antoinecarme closed 3 years ago

antoinecarme commented 7 years ago

Continuous Ranked Probability Score (CRPS) is a non-parametric (distribution-based) measure of the quality of a forecast.

As such, it should be interesting to evaluate it as the default performance measure used in PyAF for model selection (as a replacement for MAPE, which has some known issues with zero values in the signal).

Some references :

  1. https://github.com/FK83/scoringRules
  2. https://stats.stackexchange.com/questions/112250/understanding-the-rank-probability-score
  3. https://www.kaggle.com/wiki/ContinuousRankedProbabilityScore
  4. https://cran.r-project.org/web/packages/scoringRules/vignettes/gettingstarted.html
  5. MAPE/sMAPE known issues : https://robjhyndman.com/hyndsight/smape/
antoinecarme commented 6 years ago

A similar method was used in scoring for GEFCom 2014 :

https://robjhyndman.com/hyndsight/gefcom2014/

image

antoinecarme commented 4 years ago

Probabilistic Forecasting (forecasting the signal distribution/quantiles over the horizon) :

  1. Ichiro Takeuchi, Quoc V. Le, Timothy D. Sears, Alexander J. Smola, (2006), Nonparametric Quantile Estimation, Journal of Machine Learning Research 7 1231–1264 https://www.lokad.com/probabilistic-forecasting-definition

  2. Use and Communication of Probabilistic Forecasts Adrian E. Raftery University of Washington August 22, 2014 https://arxiv.org/pdf/1408.4812.pdf

  3. https://www.lokad.com/probabilistic-forecasting-definition

antoinecarme commented 3 years ago

Closing

antoinecarme commented 3 years ago

https://otexts.com/fpp3/distaccuracy.html