Error with eval_forecasts(metrics = "coverage")

[Bisaloo]

Here is a reproducible example:

library(scoringutils)
#> Note: The definition of the weighted interval score has slightly changed in version 0.1.5. If you want to use the old definition, use the argument `count_median_twice = TRUE` in the function `eval_forecasts()`

range_example_wide <- data.table::setDT(scoringutils::range_example_data_wide)
range_example <- scoringutils::range_wide_to_long(range_example_wide)

eval <- scoringutils::eval_forecasts(range_example,
                                     summarise_by = "model",
                                     quantiles = c(0.05, 0.95),
                                     metrics = "coverage",
                                     sd = TRUE)
#> Error in eval_forecasts_quantile(data = data, by = by, summarise_by = summarise_by, : object 'res' not found

^{Created on 2021-07-08 by the reprex package (v2.0.0.9000)}

This happens because the code to compute coverage assumes that a res object already exists:

https://github.com/epiforecasts/scoringutils/blob/8437a562002270fb9cd957c8cc801a27a46a0787/R/eval_forecasts_quantile.R#L79-L82

but this is only the case if interval_score is listed in metrics and this code has run before:

https://github.com/epiforecasts/scoringutils/blob/8437a562002270fb9cd957c8cc801a27a46a0787/R/eval_forecasts_quantile.R#L56-L76

There is the same issue with metrics = "coverage" for example.

Related: #77

I can submit a PR to fix this if you want but feel free to fix it directly if it's simpler for you.

[nikosbosse]

Thank you for pointing this out! I addressed this issue via https://github.com/epiforecasts/scoringutils/pull/112

One small note: the correct syntax for what you were trying to achieve is this

  range_example_wide <- data.table::setDT(scoringutils::range_example_data_wide)
  range_example <- scoringutils::range_wide_to_long(range_example_wide)

  eval <- scoringutils::eval_forecasts(range_example,
                                       summarise_by = c("model", "range"),
                                       metrics = "coverage",
                                       sd = TRUE)

which gives you one score for every range and then you need to filter for the desired ranges. The reason is that scoringutils gives you everything in a long format (which admittedly is maybe slightly confusing). The quantiles option you had selected would give you the quantiles of the score as a summary measure (so e.g. if you had 100 scores for every model, specifying quantiles would give you the desired quantiles in addition to the mean that is returned per default)

[nikosbosse]

Addressed via #112. Thanks a lot!