search

dtcenter / MET

Model Evaluation Tools
https://dtcenter.org/community-code/model-evaluation-tools-met
Apache License 2.0
77 stars 24 forks source link

Consider adding CRPS and CRPSS to the probabilistic verification in Grid-Stat and Point-Stat. #1023

Closed dwfncar closed 6 years ago

dwfncar commented 6 years ago

This issue came up in two MET-Help tickets from folks at UKMO:
   https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86037
   https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291


Currently, the ensemble-stat tool produces output for CRPS and CRPSS, but the grid-stat and point-stat tools do not when evaluating probability forecasts.


This task is to investigate how we should add them. Should they be derived from the Nx2 contingency tables from the PCT line type? Or should they be computed from the raw probability/obs pairs directly?


Tressa will write up details on how this should be done, and we'll implement that method.


Tressa has attached the Hersbach (2000) paper here, which shows how to calculate the CRPS using the empirical distribution function (e.g. histogram) instead of fitting a normal distribution. This would work on the probabilistic contingency table info. This looks like a pretty simple sum of some areas, with just some record keeping. The graphics in the paper will probably provide better guidance than some text here. [MET-1023] created by johnhg

dwfncar commented 6 years ago

John thinks we should add this to the existing PSTD line type. by johnhg

dwfncar commented 6 years ago

This issue doesn't make sense to fix. CRPS doesn't apply to probability forecasts. Instead, it applies when you compare an ensemble of forecast values to an observation value. In another ticket, I updated Point-Stat by adding the ECNT output line type when applying the HiRA verification method. by johnhg