cmu-delphi / exploration-tooling

tools for evaluating and exploring forecasters
Other
0 stars 0 forks source link

Support baseline score scaling and baseline forecaster selection #40

Closed nmdefries closed 11 months ago

nmdefries commented 11 months ago

This lets the user select which forecaster to compare others against. Currently this defaults to the first in the list, but we can change this once we have a baseline forecaster running. Unclear if it is useful to be able to normalize by different forecasters. If not, it's easy to remove that option.

Plus clean up the interface somewhat.

dsweber2 commented 11 months ago

So one complication is that each forecaster is also associated with a specific ahead, so if you try to facet by ahead, it only displays the ahead you're scaling by. As long as we ignore the current ensembles, the number at the end corresponds to the ahead. image

nmdefries commented 11 months ago

Oh, I was wondering why each forecaster only ever had one ahead. I'm used to thinking about each forecaster including multiple aheads. Is it important to have the aheads separated into different forecasters?

If not, we could combine all aheads for a given forecaster type into a single forecaster. I'm imagining combined forecasters would be used both in the baseline and "normal" forecaster selections. To look at separate aheads, we can just facet by them or put them on the x axis.

Alternately, if we want to keep aheads separated, make it so you can't normalize by an arbitrary forecaster.

dsweber2 commented 11 months ago

For parallelization reasons its best to separate by ahead, but we could add a step to combine them after the fact, so we have a indecisive.specified target that combines indecisive.specified.1, indecisive.specified.2 etc.

nmdefries commented 11 months ago

Okay, I'll add a step to combine them. I think that makes the most sense.