CODARcode / MGARD

MGARD: MultiGrid Adaptive Reduction of Data
Apache License 2.0
37 stars 25 forks source link

Add Self-Describing Format #172

Closed ben-e-whitney closed 2 years ago

ben-e-whitney commented 2 years ago

The main purpose of these commits is to implement a redesigned self-describing format for compressed datasets. Two pieces of the API, both marked experimental in the most recent release, are changed: the format of the buffer returned by CompressedDataset::data and the format expected of the buffer taken by this decompress overload:

MemoryBuffer<const unsigned char> decompress(void const *const data,
                                                  const std::size_t size);

Note that the return type of this function has also been changed.

The commits

  1. define a self-describing buffer header with Protobuf,
  2. add code to parse and check the header,
  3. modify decompress and compress to use header objects internally for configuration, and
  4. modify the CLI to use the self-describing format.
ben-e-whitney commented 2 years ago

@qliu21, you can review and merge this now.