search

cmu-delphi / forecast-eval

delphi.cmu.edu/forecast-eval
MIT License
5 stars 2 forks source link

Speed up state truth data-pulling by moving out of `evalcast` #225

Open nmdefries opened 2 years ago

nmdefries commented 2 years ago

Create evaluate_chu-like function for pulling truth data and scoring state predictions. Because the function bypasses evalcast's built-in data-fetching, this should be a lot faster.

Haven't worked on this in a long time, though, so the code will need to be reviewed, and checked for correctness and speed compared to the current pipeline. This may be entirely ready to go as-is :crossed_fingers:.

nmdefries commented 2 years ago

IIRC (it was a long time ago, though) all state truth data using this approach and using evalcast matched except for California.

nmdefries commented 2 years ago

Test run in production failed. Update: Below changes should address this.

nmdefries commented 2 years ago

After adding some fixes, a production run succeeded, with a run time of 4h 25m. This is a reduction of about 1.5h, in line with previous estimates of how long downloading truth data in evalcast takes.