jordansread
commented
2 years ago This PR moves away from the rectangle-based NLDAS driver file builds originally created in #1. What the prior solution was doing was setting a rectangle to pull NLDAS NetCDF files for all time (but chunked into files of 25k steps each), writing all of the time values to single feather files for each variable and cell (still at native hourly NLDAS resolution) and then combining the variables into a single daily .csv for each cell. This process worked fine but each time we wanted to update time or space we had to set the appropriate rectangle and build the files.
The new method I've coded works mostly the same but instead of using multiple rectangles in a mosaic covering our study area, it uses the single CONUS footprint here. The indicator file from that build now lands in our in here and the other things work mostly the same.
But, since we're no longer moving along these rectangles, which were manageable to build the files, I'm now grouping cells spatially in task plans so that chunks of cells and chunks of driver files are written per task, as opposed to a GLM .csv per task which was the old workflow and not scalable. So that's what the other main changes are. Then some clean-up I took care of when I was in there.
lindsayplatt
^ big changes after updating NLDAS to the whole CONUS.
Getting this in now because it has build/ind updates and will provide more context and explanation tomorrow.
Fixes #328 but this included some clean-up in how we were dealing with NLDAS data as specified originally in #1