The package automatically scores predictions against observed values. In order to compare model performance, some more refined techniques could be implemented such as pair-wise model comparisons and comparison within a mixed-effect model framework. Some examples of that exist in other places we can point to.
[impact: Forecasts of Covid-19 have a large influence on policy-makers. Oftentimes, however, these predictions are not evaluated thoroughly. Giving researchers the tools necessary to easily assess the accuracy of their forecasts may help drastically improve epidemiological forecasting - not only for this pandemic, but also for future outbreaks. The scoringutils package is currently used by research teams in the UK and US. It is also set to be deployed on a larger scale as part of the evaluation for the US Forecast Hub (https://github.com/reichlab/covid19-forecast-hub) which directly informs the US Centers for Disease Control and Prevention.]
[originally proposed by @nikosbosse]
[suggested repo: https://github.com/epiforecasts/scoringutils | @nikosbosse]
[additional notes: ]
The package automatically scores predictions against observed values. In order to compare model performance, some more refined techniques could be implemented such as pair-wise model comparisons and comparison within a mixed-effect model framework. Some examples of that exist in other places we can point to.
[impact: Forecasts of Covid-19 have a large influence on policy-makers. Oftentimes, however, these predictions are not evaluated thoroughly. Giving researchers the tools necessary to easily assess the accuracy of their forecasts may help drastically improve epidemiological forecasting - not only for this pandemic, but also for future outbreaks. The scoringutils package is currently used by research teams in the UK and US. It is also set to be deployed on a larger scale as part of the evaluation for the US Forecast Hub (https://github.com/reichlab/covid19-forecast-hub) which directly informs the US Centers for Disease Control and Prevention.] [originally proposed by @nikosbosse] [suggested repo: https://github.com/epiforecasts/scoringutils | @nikosbosse] [additional notes: ]