Create metadata dictionary - Githubissues

reichlab / covid19-forecast-hub

Projections of COVID-19, in standardized format

https://covid19forecasthub.org

Other

445 stars 326 forks source link

Create metadata dictionary #456

Closed jarad closed 4 years ago

jarad commented 4 years ago

Provide information on what the metadata fields mean.

jarad commented 4 years ago

First stab here 7961eebeeffcd111a1d093dfb1692a08a6eb754f

jarad commented 4 years ago

@nickreich Please search for ??? within METADATA.md and provide explanation where I was unsure.

nickreich commented 4 years ago

@jarad I removed a bunch of fields and added some clarifying text around the edges of what you already had.

nickreich commented 4 years ago

I added some text about model urls. Deleted the two existing model_repo and model_output fields and tried to consolidate in a single model_url field. Is there a better way to do this?

A url to a website that has additional data about your model. We encourage teams to submit a url to the most user-friendly version of your model, e.g. a dashboard, or similar, that displays your model forecasts. If you have additionally a data repository where you store forecasts and other model code, please include that in your methods section below. If you only have a more technical site, please include that link here.

nickreich commented 4 years ago

should we include a license field? either needs to be one of a set of selected licenses or "custom" and then they have a license file?

nickreich commented 4 years ago

Also, what about this idea of including a designated_team_model or something like this for the purposes of model ranking/evaluation?

nickreich commented 4 years ago

three separate things a model would be used for that need to be encoded in metadata somehow:

ranking against other models by scores. Teams specify one model that will be ranked. All models will be scored. teams should be able to change their ranking if model methodology is substantively changed.
ensemble consideration. The "ranked" model from each team will be eligible for the ensemble. Teams may also designate additional models that have substantively different methodological approaches for inclusion in the ensemble. Examples of "substantively different methods": Compartmental models, exponential smoothing. Examples of not "substantively different methods": SEIRD model with different priors or parameter settings. The COVID-19 Forecast Hub ensemble team will have a final say about whether the methods are substantively different enough to warrant inclusion. Each week, all eligible models will be assessed for inclusion based on current criteria.
visualization. models that are eligible for inclusion in the ensemble should be visualized

nickreich commented 4 years ago

twitter handle?

jarad commented 4 years ago

license was discussed in #342

hannanabdul55 commented 4 years ago

we also had discussed earlier to include a field to indicate which models are included in the visualization. Could that also be added here?

jarad commented 4 years ago

if team_model_designation is "primary" or "secondary" then the model should be included in the visualization

jarad commented 4 years ago

[dictionary[(https://github.com/reichlab/covid19-forecast-hub/blob/master/data-processed/METADATA.md)