search

superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.78k stars 89 forks source link

LTX Compression #132

Closed benbjohnson closed 1 year ago

benbjohnson commented 1 year ago

LTX files are currently sent to replicas as uncompressed blobs. LTX files should support compression internally.

benbjohnson commented 1 year ago

Update: HTTP compression was attempted in #155 but it interfered with http.Flusher. Implementing in LTX would allow us to compress once instead of compressing for each replica connection.

buzinas commented 1 year ago

For this type of transfer, zstd might be a good candidate!

benbjohnson commented 1 year ago

I used lz4 on Litestream and it worked pretty well. From my understanding, lz4 is faster but zstd has a better compression ratio. I've found that the compression & decompression speed is usually more important for something like LiteFS rather than eeking out every last byte of compression.

benbjohnson commented 1 year ago

Closed via #249