Our initial use case of weather-sp was to split monolithic files by variable. Now, we've encountered a case where we'd like to split atmospheric data by model level.
To better generalize to all future uses of the splitter, let's re-design the tool to support arbitrary divisions of the data hypercube. This, ultimately, would help us improve usage of the weather-dl tool: If we didn't have to worry about the final file structure (because we could make it uniform), we could better optimize downloads. Specifically, we can get as much data as we can under the memory limit, which is ~75 GB for MARS as of writing.
uhager
commented
2 years ago
Our initial use case of
weather-spwas to split monolithic files by variable. Now, we've encountered a case where we'd like to split atmospheric data by model level.To better generalize to all future uses of the splitter, let's re-design the tool to support arbitrary divisions of the data hypercube. This, ultimately, would help us improve usage of the
weather-dltool: If we didn't have to worry about the final file structure (because we could make it uniform), we could better optimize downloads. Specifically, we can get as much data as we can under the memory limit, which is ~75 GB for MARS as of writing.