search

epiforecasts / scoringutils

Utilities for Scoring and Assessing Predictions
https://epiforecasts.io/scoringutils/
Other
48 stars 21 forks source link

coverage of function factory #907

Open seabbs opened 2 months ago

seabbs commented 2 months ago

As mentioned in https://github.com/hubverse-org/hubEvals/pull/46 do we want to have any kind of special metric function factories for generating things like multiple interval coverages?

I think the answer is potentially with the caveat of do we want a helper across metrics. (i.e duplicate this metric for this arg and the arg input is a vector) or do we want just nicely named special cases.

nikosbosse commented 2 months ago

Could be a nice idea

seabbs commented 2 months ago

Any thoughts on implementation questions? I am happy to have a first go at this

nikosbosse commented 2 months ago

My initial thought is that interval coverage is currently the only case that comes to mind where we'd use this. Do you have other cases in mind?

I like the input vector mechanic.

I think it would be cool to make it such that it works nicely with get_metrics - not sure how though.

nikosbosse commented 2 months ago

Aside thoughts:

do we actually want to export select_metrics? It does the same as the exclude and select args.

We had discussed other tools to compose metrics, e.g. selecting only the wis metrics or something like that. O wasn't really clear what that should look like back then. Bumping it up in case it might be relevant here

seabbs commented 2 months ago

My initial thought is that interval coverage is currently the only case that comes to mind where we'd use this. Do you have other cases in mind?

Yes its the only one I can think of as well. We could literally just add add arg to generate new functions to the quantile get_metrics method?

do we actually want to export select_metrics? It does the same as the exclude and select args.

Not sure this is for here but can discuss. I favour keeping it as I prefer the workflow of chaining simple functions vs overloading get_metrics. If anything I would get rid of their internal ability to select...