Open scottstanie opened 4 months ago
The size is ~half when we truncate the mantissa bits of more floating point rasters.
448M Mar 5 18:12 original.nc 348M Mar 5 18:09 t2.nc # truncate troposphere 304M Mar 5 18:10 t3.nc # truncate ionosphere 228M Mar 5 18:12 t4.nc # truncate unwrapped_phase
The corrections layers certainly do not need the full float32 precision; the unwrapped_phase precision is more up for discussion.
Another view at this using this script
Compression Ratios for HDF5 File: output/forward/20221107_20230729.unw.nc ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Dataset ┃ Storage Size (MiB) ┃ Compression Ratio ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ /connected_component_labels │ 1.83 │ 68.124 │ │ /corrections/ionospheric_delay │ 43.80 │ 5.699 │ │ /corrections/plate_motion │ 0.37 │ 670.321 │ │ /corrections/solid_earth_tide │ 0.37 │ 670.321 │ │ /corrections/tropospheric_delay │ 93.91 │ 2.658 │ │ /interferometric_correlation │ 74.09 │ 3.369 │ │ /persistent_scatterer_mask │ 1.05 │ 59.218 │ │ /temporal_coherence │ 72.81 │ 3.429 │ │ /unwrapped_phase │ 132.41 │ 1.885 │ │ <total chunked datasets> │ 420.66 │ 4.599 │
after
Compression Ratios for HDF5 File: t4.nc ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Dataset ┃ Storage Size (MiB) ┃ Compression Ratio ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ /connected_component_labels │ 1.83 │ 68.124 │ │ /corrections/ionospheric_delay │ 1.20 │ 207.574 │ │ /corrections/plate_motion │ 0.37 │ 670.321 │ │ /corrections/solid_earth_tide │ 0.37 │ 670.321 │ │ /corrections/tropospheric_delay │ 13.34 │ 18.719 │ │ /interferometric_correlation │ 74.09 │ 3.369 │ │ /persistent_scatterer_mask │ 1.05 │ 59.218 │ │ /temporal_coherence │ 72.81 │ 3.429 │ │ /unwrapped_phase │ 55.82 │ 4.472 │ │ <total chunked datasets> │ 220.89 │ 8.758 │ └─────────────────────────────────┴────────────────────┴───────────────────┘
The size is ~half when we truncate the mantissa bits of more floating point rasters.
The corrections layers certainly do not need the full float32 precision; the unwrapped_phase precision is more up for discussion.
Another view at this using this script
after