search

lemire / FastPFor

The FastPFOR C++ library: Fast integer compression
Apache License 2.0
883 stars 124 forks source link

Looking for software for delta compression #106

Open christophgil opened 1 year ago

christophgil commented 1 year ago

Hi Lemire,

I found your software when searching for a solution regarding large binary data.

We are using files with an unpublished proprietory file format. With hexdump I noticed that it belongs of long data blocks which could efficiently compressed with delta-compression techniques. It is 4 bytes each. Longs or floats (mantissa+exp) I do not know yet.

Do you know of a compression programs that could compress such file. It would need to identify these block positions automatically. Could this be done with FastPFor?

Many thanks

Christoph

lemire commented 1 year ago

We do have delta compression routines in FastPFor, yes.