search

LLNL / zfp

Compressed numerical arrays that support high-speed random access
http://zfp.llnl.gov
BSD 3-Clause "New" or "Revised" License
768 stars 155 forks source link

Deserializing const_array #228

Open systemed opened 7 months ago

systemed commented 7 months ago

I'm (belatedly!) moving from zfp 0.5.x to 1.0.1 and am very impressed with the read-only variable rate arrays.

With the fixed-rate arrays it was possible to serialize to disk once and then read them in later, thereby saving time compressing them on application startup.

Is there any way to do this with const_array? The documentation only mentions "zfp’s read-write compressed arrays can be serialized to sequential, contiguous storage".

I'm not too worried about the headers (I can store that information independently) but being able to populate compressed_data from disk would be really useful.

lindstro commented 7 months ago

I'm afraid we didn't have time to work out (de)serialization with the const_array classes for the 1.0.x releases, but it is likely that this will be supported in the next release. We have to deal with the same issue to support parallel variable-rate decompression, where we need to quickly locate any given block in the compressed stream. We're essentially using the same block offset coding strategy for that. Because we cannot embed these block offsets in the compressed stream while retaining backwards compatibility, we need for the block index to be stored separately, and we're currently working on support for (de)serializing this auxiliary data.

systemed commented 7 months ago

That's great to know - thank you. I'll go with compressing at application startup for now, but move to deserialization when that's available.