search

rosepearson / GeoFabrics

A package for generating hydrologically conditioned DEMs and roughness maps from LiDAR and other infrastructure data. Check the wiki for install and usage instructions, and documentation at https://rosepearson.github.io/GeoFabrics/
GNU General Public License v3.0
25 stars 11 forks source link

Compress output NetCDF files #236

Closed CyprienBosserelle closed 4 months ago

CyprienBosserelle commented 5 months ago

using a deflation on Netcdf files will reduce file size by hopefully a huge factor:

https://gis.stackexchange.com/questions/377299/adding-compression-to-a-netcdf-file-using-xarray

rosepearson commented 5 months ago

Thanks @CyprienBosserelle

It looks like I will want to do something like:

 encode = {"data_source":  {'zlib': True, 'complevel': 1, 'grid_mapping': dem.data_source.encoding['grid_mapping'], },
                   "lidar_source":  {'zlib': True, 'complevel': 1, 'lidar_source': dem.lidar_source.encoding['grid_mapping'], },
                   "z":  {'zlib': True, 'complevel': 1, 'z': dem.data_source.encoding['z'], }, 
                   "zo":  {'zlib': True, 'complevel': 1, 'zo': dem.data_source.encoding['zo'], }}
 dem.to_netcdf(filename, encoding=encode, format="NETCDF4", engine="netcdf4")
rosepearson commented 4 months ago

Note also need to ensure the spatial info is written into the compression as otherwise no spatial information is included. Updated example above