epiforecasts / scoringutils

Utilities for Scoring and Assessing Predictions
https://epiforecasts.io/scoringutils/
Other
47 stars 19 forks source link

Should `score()`/`apply_metrics()` return warning about failed metrics in a batch? #811

Open jamesmbaazam opened 2 months ago

jamesmbaazam commented 2 months ago

Should we also print the warnings in a batch instead of eagerly at the time of occurrence? They could easily get lost in the other warnings as the functions are getting executed.

I imagine a setup where at the time of printing the successful scores dt, we also print "In addition, the following metrics failed with the following warning". This is nicely discussed in this blog post about deferring errors. Worth considering.

_Originally posted by @jamesmbaazam in https://github.com/epiforecasts/scoringutils/pull/803#discussion_r1592175575_

seabbs commented 1 month ago

I like this idea