epiforecasts / scoringutils

Utilities for Scoring and Assessing Predictions
https://epiforecasts.io/scoringutils/
Other
48 stars 21 forks source link

Handling case in WIS where quantile levels don't form valid intervals #943

Closed nikosbosse closed 1 month ago

nikosbosse commented 1 month ago

Description

This PR closes #926.

As discussed in #926, this PR lets wis() error by default if not all quantile levels form valid prediction intervals. I realised that there was a na.rm argument already (it was just set to TRUE). I think FALSE is a better default. If the quantile levels aren't symmetric, the function errors immediately.

(Note that there is a second way in which na.rm can be relevant: if a specific quantile is missing for a forecast. With na.rm = TRUE NA values will be dropped and otherwise the output is NA.)

Checklist

seabbs commented 1 month ago

Kätzchen spielt im Hof Miau, miau, ruft es leise Mutters Strickzeug tanzt

seabbs commented 1 month ago

Note that there is a second way in which na.rm can be relevant: if a specific quantile is missing for a forecast. With na.rm = TRUE NA values will be dropped and otherwise the output is NA.)

This was the motivation for the previous setting wasn't it because historically hubs allowed a mixed set of quantiles to be submitted and we wanted to support that?

seabbs commented 1 month ago

Alte Katze schläft Erinnerung an Spiele Strickzeug längst verblasst

nikosbosse commented 1 month ago

Kätzchen spielt im Hof Miau, miau, ruft es leise Mutters Strickzeug tanzt

I couldn't agree more.

This was the motivation for the previous setting wasn't it because historically hubs allowed a mixed set of quantiles to be submitted and we wanted to support that?

This is really more missing forecast values. If you had mixed sets of quantiles, then you'd get a warning in score(), but otherwise it should be fine.

@elray1 @nickreich are there any issues you'd expect from this PR? TLDR: wis() errors if you supply quantile_levels like c(0.1, 0.4, 0.7) that don't form intervals. And if you have quantile_levels = c(0.1, 0.2, 0.8, 0.9) but predicted = c(2, 4, 8, NA) then it will result in NA by default.

seabbs commented 1 month ago

This is really more missing forecast values. If you had mixed sets of quantiles, then you'd get a warning in score(), but otherwise it should be fine.

Okay makes sense.

nikosbosse commented 1 month ago

Also, I have found a great use of AI. See if you can guess what the prompt was.

1950's German folk poetry? :D

seabbs commented 1 month ago

haiku from a 19th century german 9 year old and a haiku from the same 9 year old when they are 95