search

rosepearson / GeoFabrics

A package for generating hydrologically conditioned DEMs and roughness maps from LiDAR and other infrastructure data. Check the wiki for install and usage instructions, and documentation at https://rosepearson.github.io/GeoFabrics/
GNU General Public License v3.0
29 stars 11 forks source link

performance improvement - separate out raster generation for each waterway #262

Closed rosepearson closed 1 month ago

rosepearson commented 1 month ago

Currently we produce a single waterway raster for all waterways considered together. This leads to errors with large files. consider generating each separately as generally these will be much smaller than the overall waterway network. This should simplify memory management and reduce errors with files failing to be saved.

Actions:

rosepearson commented 1 month ago

Note failing CI seems related to issues specific to netcdf4 1.7.1 on https://github.com/Unidata/netcdf4-python/issues?q=is%3Aissue%20state%3Aopen%20hdf%20error

This si something @jennan ran into recently as referenced in this commit - https://git.niwa.local/forecasting-services/packages/highres_nwm/-/commit/f45845f7a19739f73c94d920e6acd86301d60020 he migrated to h5netcdf and also builds his environments using pip.

rosepearson commented 1 month ago

Further notes on CI errors related to saving / loading rasters:

Comparing the last successful CI tests: https://github.com/rosepearson/GeoFabrics/actions/runs/10825077248/job/30033393178 against the first failing tests: https://github.com/rosepearson/GeoFabrics/actions/runs/10839705758/job/31267478081

Library versions (All conda installed): netcdf4 - 1.7.1 - same libnetcdf - 4.9.2 - same libgdal-hdf5 - 3.9.2 - same libgdal-netcdf - 3.9.2 - same libgdal-hdf4 - 3.9.2 - same hdf4 - 4.2.15 - same hdf4 - 1.14.3 - same rioxarray - 0.17.0 same rasterio - 1.3.11 - same