Current pipeline writes to a files_missing folder where we can see which outputs were expected from the config and weren't able to be generated. This currently include 216 combinations of forecast dates and locations. Many of these are places where the input_ww_data is just a df of 0 rows, but others are places with wastewater data that will run locally.
Tasks
[x] location-forecast-dates without wastewater should not be included in this files_missing list. These are expected missing files.
[x] rerun the remaining failed jobs using larger RAM VMs. If that doesn't work, consider manually downloading necessary raw files to run locally....
kaitejohnson
commented
2 months ago
Goal
Current pipeline writes to a
files_missingfolder where we can see which outputs were expected from the config and weren't able to be generated. This currently include 216 combinations of forecast dates and locations. Many of these are places where theinput_ww_datais just a df of 0 rows, but others are places with wastewater data that will run locally.Tasks
files_missinglist. These are expected missing files.