search

cmu-delphi / forecast-eval

delphi.cmu.edu/forecast-eval
MIT License
5 stars 2 forks source link

Stop calculating deaths and cases scores #307

Closed nmdefries closed 11 months ago

nmdefries commented 12 months ago

Closes #248.

Only hospital admissions predictions are still active; cases and deaths forecasts are no longer supported by Forecast Hub and the JHU truth data used to evaluate these is no longer being updated. This means that historical error metrics for cases and deaths are fixed, so we no longer need to calculate them.

The pipeline is still able to recalculate cases and deaths scores if we wish -- this only requires updating the signals vector. To make sure that these archived scores are still available for the dashboard, the relevant score RDS files are downloaded from the S3 bucket and then re-uploaded unmodified after the pipeline completes.

dsweber2 commented 12 months ago

Oh, another thought from @dshemetov's work on the old exploration scripts, as far as saving memory is concerned, we may want to sprinkle gc() around, since it isn't getting called by R as often as we'd like.

nmdefries commented 11 months ago

On the gc()s, any suggestions on how frequent they should be? We have just a couple in here already, but there are also some long loop cycles (e.g. while downloading forecasts).

dsweber2 commented 11 months ago

iirc it was around get_covidhub_predictions that I OoMed, so it might actually belong in the for loops there. In the cases where Dmitry got a large decrease in memory, there wasn't even explicit rms or anything like that.

nmdefries commented 11 months ago

Just a very light dusting of gc() added here