lindstro
commented
1 year ago @markcmiller86 This looks interesting but I'm not sure how to best expose zfp through such an API since zfp fundamentally operates on multidimensional arrays, whereas the API seems geared toward streams of numbers. The I/O stream would somehow have to know about array dimensions and/or strides for this to work well. There's a similar problem of specifying compression settings, e.g., error tolerance. I've not had a chance to take a close enough look at the I/O library to see if there's an easy way to incorporate that. Perhaps one could use something similar to the C++ stream manipulators like std::set_precision() to specify compression settings and array metadata.
The only other alternative I can think of is to have the user reorganize the data into contiguous blocks, but even then zfp wouldn't know what the dimensionality of the data is (1D, 2D, 3D, ...).
I happened to run across this C/C++ stream io utility suite which apparently already integrates zlib, gzip and zstd and I am thinking it could be a potential place to integrate ZFP as well. I suspect they are doing only byte-level compression in their implementations and not doing numeric and/or array-level compression but I've always wondered what it would take (either in the way of stream manipulators or whatever) to extend C++ iostream interface to support ZFP compression.
So, this isn't really an issue...just a bookmark for possible future work.