epiforecasts / scoringutils

Utilities for Scoring and Assessing Predictions
https://epiforecasts.io/scoringutils/
Other
48 stars 20 forks source link

Issue #761 - Update input checks and docs for `quantile_score()` #768

Closed nikosbosse closed 4 months ago

nikosbosse commented 4 months ago

Description

This PR closes #761.

Previously, the function quantile_score() inherited its arguments from wis(), but the actual format it expected was different from the one used by the other quantile-based metrics. The function used to expect a vector of observed values and a single vector with predicted values and a single quantile level. This is in contrast to the other quantile-based functions, which usually expect a matrix of predicted values and are also able to accept several quantile levels.

This PR changes the expected input format of quantile_score() such that it conforms to what the other functions accept. Instead of computing the quantile score for a single vector, it now computes the score for several quantile forecasts and returns an average score. One implication of this is that the output of wis() and quantile_score() are now mostly identical, apart from the fact that

The PR

I haven't added a news item because the update is changing the input formats for all functions (we just hadn't properly implemented that change for quantile_score() yet)

Checklist

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.29%. Comparing base (84cbcc9) to head (96c1010).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #768 +/- ## ======================================= Coverage 95.29% 95.29% ======================================= Files 21 21 Lines 1594 1595 +1 ======================================= + Hits 1519 1520 +1 Misses 75 75 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.