search

cmu-delphi / forecast-eval

delphi.cmu.edu/forecast-eval
MIT License
5 stars 2 forks source link

When changing score to/from coverage, show main plot and make sure truth data is relevant #270

Closed nmdefries closed 1 year ago

nmdefries commented 1 year ago

Description

  1. The default behavior in the input$scoreType observer is to reuse the existing truth plot. This is valid as long as the locations reported by each score type are the same. Most of the time, that's true; WIS, absolute error, and sharpness/spread all use the selected location.

    However, regardless of which location is selected, coverage aggregates scores over states that have forecasts reported by all selected forecasters. This means that when changing to or from "coverage" and any single state or "US" is selected as the location (that is, we are not aggregating over shared locations), the displayed truth data will be incorrect if we just show the pre-existing plot. So we want to regenerate the truth plot by setting the flag that controls that behavior, USE_CURR_TRUTH, to FALSE in this case.

  2. The old code was calling updateAsOfData() in some cases (only when the selected location was "US" -- what about other single locations?). We do need to do that to make sure the truth data is relevant, either invalidating cached as-of truth data or fetching new as-of truth data.

    However, when we're not requesting as-of truth data, updateAsOfData() also sets RE_RENDER_TRUTH to TRUE. This flag causes summaryPlot() to not generate a new main plot showing forecaster scores. We need to reset RE_RENDER_TRUTH to FALSE.

Fixes

Logan described the following issue on Slack:

Somehow by selecting a few teams... I reached a mode where switching from WIS to coverage would not plot on the main tab, but I could switch back to WIS and it would again show the WIS plot, and I could switch to the archive tab, switch to coverage, switch back to main tab, and coverage plots would show. And the same thing going from coverage to WIS plots. ... If you have US selected then that triggers the non-plotting behavior