Add DEFLATE compression

ept commented 3 years ago

This PR adds support for DEFLATE compression. As explained in #317, compression can be used in two places:

On an individual change, the whole change can be compressed. The change hash is computed on the uncompressed change, and the checksum included in the change is based on the uncompressed change. We do this because two different compressors given the same input do not necessarily produce the same compressed byte stream.
On a compressed document, one column at a time can be compressed. I changed the definition of the columnId to reserve one bit to indicate whether the column is compressed. We do this so that it is possible to load only a subset of columns of a document.

Compression is only enabled when the data is larger than some threshold (currently 256 bytes) since it's not worth it on very short values.

@alexjg could I leave the implementation on the Rust side with you?

alexjg commented 3 years ago

Looks pretty straightforward, I'll pick it up for the Rust implementation later this week.

alexjg commented 3 years ago

The Rust implementation of this is ready to merge now.

ept commented 3 years ago

@alexjg Excellent, thank you! I have added some JS-wasm interop tests to check that deflate-compressed changes and documents work correctly in both directions. Merging this into performance now.

automerge / automerge-classic

Add DEFLATE compression #330