search

DOI-USGS / ale

Abstraction Layer for Ephemerides (ALE)
Other
13 stars 33 forks source link

Enhancement: Support for ISD Compression #604

Closed jlaura closed 5 months ago

jlaura commented 6 months ago

When working with a large number of ISDs, the total data volume can start to become an issue. To combat this, some sort of compression for ISDs would be terrific. Right now, this is living in user land where one can compression/uncompress. This is to request a potential enhancement in ALE to standardize on one compression method. The idea being that down stream ISD consumers could choose to support compressed ISDs because they know what the compression is.

This review of JSON suggests that simple brotli compression is both fast and efficient. It also seems that brotli is well supported.

thareUSGS commented 6 months ago

I like that brotli was designed for http streaming and there is a C++ header-only option (https://github.com/NewYaroslav/brotli-hpp). Perhaps the "usgs_csm_test" helper app could include a built-in compress/decompress method so that it can be made human-readable. Well, that app would need to support decompression since there is a method to inject a camera state into a GXP "sup" supplemental file. The sup file would require the ASCII version.