Supporting additional compression formats

Nullus157 / async-compression

Adaptors between compression crates and Rust's async IO types

https://docs.rs/async-compression

Apache License 2.0

408 stars 80 forks source link

Supporting additional compression formats #12

Open fairingrey opened 5 years ago

fairingrey commented 5 years ago

Just an idea that was fishing around in my head lately, but I think it would be awesome to support additional crates for use between the newest async types/traits.

One in particular that stands out to me (and could be useful for the near future) is the zstd crate. In particular, we could apply the same kind of byte-stream encoding/decoding logic that we use for Deflate/Zlib to the zstd raw compresser/decompressers here. Also another thing to note is while it does support AsyncRead/AsyncWrite through a feature gate tokio-io, that is still on futures 0.1.

For zstd, it shouldn't take too long -- I might actually start work on it soon!

fairingrey commented 5 years ago

Now that Zstd is supported over streams, it'd be nice to aggregate some resources for anyone wishing to provide more support over different encoding schemes.

Here's a list of official content encodings for HTTP compression by the IANA registry: https://www.iana.org/assignments/http-parameters/http-parameters.xml#http-parameters-1
And a list of unofficial/non-standard encodings on Wikipedia: https://en.wikipedia.org/wiki/HTTP_compression#Content-Encoding_tokens

This likely won't be worked on unless there's a hotter compression algorithm in town I suppose, but it's here for anyone that wants to give one of them a try.

PSeitz commented 1 year ago

Async support has been requested for lz4_flex https://github.com/PSeitz/lz4_flex/issues/15. It's not a http compression encoding though.

droundy commented 1 year ago

I'll point out that lz4 is the default compression for clickhouse over http, which would make lz4 support very appealing to me.