search

NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a user, I want to compare ensemble forecasting systems with an adjustment for ensemble size #93

Open epag opened 3 months ago

epag commented 3 months ago

Author Name: James (James) Original Redmine Issue: 108992, https://vlab.noaa.gov/redmine/issues/108992 Original Date: 2022-10-11

Given an evaluation that compares two ensemble forecasting systems with different ensemble member counts When I compare them with ensemble/probabilistic scoring rules Then I want to adjust those scores for ensemble size So that I can compare them independently of ensemble size

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2022-10-11T11:49:31Z

This has arisen in a couple of contexts recently. First, #103897-156. Second, in the context of a PSL meeting where comparisons between ensemble forecasting systems were discussed for extreme events and it was noted that an adjustment for ensemble size can have a dramatic effect on any comparison/skill (and that it would be useful to apportion the difference to underlying skill vs. ensemble member count). These adjustments are available in the literature for the BS and CRPS, for example. It would need a parameter on each score.