epiforecasts / scoringutils

Utilities for Scoring and Assessing Predictions
https://epiforecasts.io/scoringutils/
Other
47 stars 19 forks source link

Issue #604 - Add support for nominal forecasts #837

Open nikosbosse opened 1 month ago

nikosbosse commented 1 month ago

Description

This PR closes #604.

Nominal forecasts are forecasts for outcomes that can fall in one of several unordered categories. This PR implements support for nominal forecasts (see #604, #607, and #608).

Specifically, the PR

Note: Throughout the process, I noticed that sadly, scoringutils is currently not "easily extensible"... To make this go smoothly, there are quite a few hoops. Some of this will be simplified in the future when we implement a separate as_forecast_nominal() function instead of a single as_forecast() function that has to do all the guesswork.


Still missing (likely for a future PR)

One current code example:

# remotes::install_github("epiforecasts/scoringutils@multiclass")
library(dplyr)
library(hubExamples)
library(scoringutils)

pred <- hubExamples::forecast_outputs |> filter(output_type == "pmf")
obs <- hubExamples::forecast_target_observations |> 
  dplyr::filter(output_type == "pmf")
hubex <- dplyr::full_join(pred, obs)

hubex |> 
  dplyr::group_by(model_id, location, reference_date, horizon, target_end_date, target, output_type) |>
  dplyr::mutate(
    observation = output_type_id[observation == 1], 
    observation = factor(observation, levels = c("low", "moderate", "high", "very high")), 
    output_type_id = factor(output_type_id, levels =  c("low", "moderate", "high", "very high"))) |>
  as_forecast(
    model = "model_id", observed = "observation", 
    predicted = "value", predicted_label = "output_type_id"
  ) |> 
  score()

Checklist

codecov[bot] commented 4 weeks ago

Codecov Report

Attention: Patch coverage is 73.62637% with 24 lines in your changes missing coverage. Please review.

Project coverage is 94.25%. Comparing base (b58434e) to head (3ba8a46). Report is 19 commits behind head on main.

Files Patch % Lines
R/score.R 0.00% 16 Missing :warning:
R/default-scoring-rules.R 0.00% 4 Missing :warning:
R/check-inputs-scoring-functions.R 92.00% 2 Missing :warning:
R/forecast.R 96.15% 1 Missing :warning:
R/metrics-nominal.R 87.50% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #837 +/- ## ========================================== - Coverage 95.53% 94.25% -1.28% ========================================== Files 21 22 +1 Lines 1569 1655 +86 ========================================== + Hits 1499 1560 +61 - Misses 70 95 +25 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.