CIROH-UA / NGIAB_data_preprocess

Tools to subset hydrofabrics, generate forcings, create default realizations for NGIAB

5 stars 2 forks source link

Zarr speedup #12

Closed JoshCu closed 3 months ago

JoshCu commented 3 months ago

Issues Fixed

3
Forcing generation now 2000x faster (ram permitting)
parallel partitions for ngen run would collide causing multiple threads to write to the same file
way way way too many generated files mixed in with important files like troute config and realization

Changes

zarr headers fetched on flask app launch
zarr headers cached on first fetch
downloaded data now always saves locally
local data is checked to see if the time range is correct, if not delete it and download new data
forcing weights calculated with exact_extract, then forcings computed with numba
updated cli to work without needing the file containing wbs
ngen outputs redirected to subfolders
catchment config for cfe moved to subfolder

Added

partition generator for use with multiprocessing in NGIAB