search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.16k stars 3.57k forks source link

Support brotli compression algorithm #11471

Open skyguard1 opened 3 years ago

skyguard1 commented 3 years ago

Brotli is a high performance, high compression ratio compression algorithm. Does pulsar plan to support this compression algorithm? If yes, I can provide a pr, see https://github.com/google/brotli, thanks

hpvd commented 3 years ago

...It's all about finding the best balance (most cost-efficient) between storage/network/compute for the use case: save storage and network bandwidth (e.g. factor 2-3) and pay with a little compute (e.g. 2-10%)

pulsar today supports LZ4, ZLIB, ZSTD, SNAPPY see https://pulsar.apache.org/docs/en/concepts-messaging/#compression

just some interesting inside from competition, including benchmarks of different algorithms https://blog.cloudflare.com/squeezing-the-firehose/

hpvd commented 3 years ago

opened a related issue regarding doc of compression feature in pulsar: https://github.com/apache/pulsar/issues/11489

codelipenghui commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.