reichlab / forecast-repository

Codebase for Zoltar forecast repository
https://zoltardata.com/
GNU General Public License v3.0
6 stars 3 forks source link

move score retrieval from file-based to query-based (like forecast data) #245

Closed matthewcornell closed 4 years ago

matthewcornell commented 4 years ago

Our current scheme for creating score csv files for downloading score data is showing its limitation with respect to scalability by using too much server memory. Nick told me about https://github.com/reichlab/covid19-forecast-hub/issues/1091 , which will definitely break the current csv file-based approach. He has proposed moving from files to score querying, similar to how forecast querying works, as documented at https://docs.zoltardata.com/queryformat/ , an idea I really like. I propose an API that's passed a query list/dict (R/Python) similar to forecasts:

{"models": [150, 237],
 "units": [335],
 "targets": [1894, 1897],
 "timezeros": [739, 738],
 "scores": ["error", "pit"]
}

Here I show IDs for the first four items, but we'd provide a utility to convert from strings to them, as we've done with forecast queries. Notes: