Closed LacombeLouis closed 4 months ago
Is your feature request related to a problem? Please describe. I would like to calculate the Winkler Interval score. So that it can be used in the kaggle Prediction interval competition I: Birth weight competition.
Describe the solution you'd like (link)[https://www.kaggle.com/datasets/carlmcbrideellis/winkler-interval-score-metric] Model performance evaluation: The Mean Winkler Interval score (MWIS) We can assess the overall performance of a regression model that produces prediction intervals by using the mean Winkler Interval score [1,2,3] which, for an individual interval, is given by:
where 𝑦 is the true value, 𝑢 it the upper prediction interval, 𝑙 is the lower prediction interval, and 𝛼 is (1-coverage). For example, for 90% coverage, 𝛼=0.1.
Note that the Winkler Interval score constitutes a proper scoring rule [2,3].
[1] Robert L. Winkler "A Decision-Theoretic Approach to Interval Estimation", Journal of the American Statistical Association, volume 67, pp. 187-191 (1972) [2] Tilmann Gneiting and Adrian E Raftery "Strictly Proper Scoring Rules, Prediction, and Estimation", Journal of the American Statistical Association, volume 102, pp. 359-378 (2007) (Section 6.2) [3] Jonas R. Brehmer, and Tilmann Gneiting "Scoring interval forecasts: Equal-tailed, shortest, and modal interval", Bernoulli volume 27 pp. 1993-2010 (2021)
Is your feature request related to a problem? Please describe. I would like to calculate the Winkler Interval score. So that it can be used in the kaggle Prediction interval competition I: Birth weight competition.
Describe the solution you'd like (link)[https://www.kaggle.com/datasets/carlmcbrideellis/winkler-interval-score-metric] Model performance evaluation: The Mean Winkler Interval score (MWIS) We can assess the overall performance of a regression model that produces prediction intervals by using the mean Winkler Interval score [1,2,3] which, for an individual interval, is given by:
where 𝑦 is the true value, 𝑢 it the upper prediction interval, 𝑙 is the lower prediction interval, and 𝛼 is (1-coverage). For example, for 90% coverage, 𝛼=0.1.
Note that the Winkler Interval score constitutes a proper scoring rule [2,3].
[1] Robert L. Winkler "A Decision-Theoretic Approach to Interval Estimation", Journal of the American Statistical Association, volume 67, pp. 187-191 (1972) [2] Tilmann Gneiting and Adrian E Raftery "Strictly Proper Scoring Rules, Prediction, and Estimation", Journal of the American Statistical Association, volume 102, pp. 359-378 (2007) (Section 6.2) [3] Jonas R. Brehmer, and Tilmann Gneiting "Scoring interval forecasts: Equal-tailed, shortest, and modal interval", Bernoulli volume 27 pp. 1993-2010 (2021)