human judgement ensemble model feature

[matthewcornell]

As discussed elsewhere and initially documented here: Spec doc for human judgment ensemble. Follow up questions discussed on this Slack thread.

updates:

• (for simplicity I'm leaving tasks here in this issue rather than making separate issues for each)
• 2023-03-20: "beta" version is implemented and being used on Zoltar
• 2023-04-18: continuing to work through below items - see In-process and Done

Remaining work:

In-process

• TBD

From matt

• ~ REQ: #downloadUserEnsemble" click(): show progress to user
  1. server contacted
  2. waiting for file
  3. file downloaded
  4. file saved

From Nick's 2023-04-05 reply in this thread:

• REQ: change ensemble method (there are a few simple computational methods that could be hard-coded in, in addition to the “mean” approach that it is using right now)

• BUG: there also seemed to be weirdness about switching to a target with no truth data and then switching back to one where there was truth data, then the axes got screwed up

• REQ: set up zoltar project-level viz customizations to specify which targets to include

  • "For many of the targets we don’t have any truth data (e.g. day ahead incident hospitalizations, for one) or any/many forecasts (day ahead cumulative deaths, for one)."
  • see: https://github.com/reichlab/forecast-repository/issues/347

Done

• REQ: could a user specify a minimum (or exact) number of targets each model should have (in my case, I’d want 4) and then only if a model had less than 4 would that error happen, and the user ensemble would be made for the targets for which every selected model had a forecast?
  • "The first set of models I selected had the issue where they had differing target_end_dates because one model (COVIDhub-baseline) had 8 weeks and the others had 4. A reasonable user might want to include the baseline a lot of the time, and it often goes out further than other models. Could we figure out an easy way to make this work?"
    • "For the error in bullet 3: maybe another customization could be that we want a specific number of required timezeroes, e.g. 4. So that it one model as 8 and another has 4 that’s ok because they both have 4 and we could compute the ensemble for 4 timezeroes. But if one has 2 then that model would be dropped."
    • "not all forecasts had the same target_end_dates"
• REQ: When the file is downloaded, could it have a format like: [forecastdate]-[user-modelname].csv?
  • where [forecastdate] is YYYY-MM-DD format of the timezero of the forecasts shown
• REQ: In a downloaded file, could the model column have values of [user-modelname] ?
• BUG: change server calcuation to use reference_date instead of exact timezeros to avoid empty download csv files. from @nick in this slack thread:
  • It is possible for a model to make predictions on multiple timezeroes that correspond to the same reference date. Although I think in practice this is quite rare. (It was more common early in the pandemic.) But probably something we should assume “could” happen.
  • In cases where this does happen, the latest timezero should be used in the calculation.

• 24

• REQ: change alert() calls to use bootstrap
• REQ: Is there a more obvious way to indicate if there is an error? Rather than just greying out the user ensemble?
  • E.g. adding a :warning: logo next to the User Ensemble?

• 13

• REQ: add an “action” to rename the User Ensemble based on user input?
  • ideally, enforcing at least a length limit similar to the hub and no spaces in the name, if not the [teamname}_[modelname] specification
  • Let’s call this the [user-modelname] field below.
  • This is the name that should also be displayed in the “Select Models” menu.
• ISSUE: more cleanly handle user model name conflict

[matthewcornell]

I believe this is done.