search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.82k stars 1.34k forks source link

Add ZSTD compression support to forward plugin #4162

Open zamazan4ik opened 1 year ago

zamazan4ik commented 1 year ago

Is your feature request related to a problem? Please describe.

ZSTD is a modern compression algorithm developed by Facebook. Using ZSTD has one of the best compression/resource consumption ratios in the compression area.

Describe the solution you'd like

Have a ZSTD compression support in additoin to GZip. Would be awesome, if will be a possibility to specify the compression level as well.

Describe alternatives you've considered

Left all things as is.

Additional context

Maybe this change will require changing FluentBit implementation as well.

Garfield96 commented 1 year ago

I would also like to see support for ZSTD. While the compression ratio of gzip is often decent, it is slow, which is a problem for Fluentd due to the GIL of CRuby. Apart from performance, gzip combined with TLS can be a security issue (cf. https://en.wikipedia.org/wiki/BREACH).

While there are other good compression algorithms (e.g. LZ4 or snappy), ZSTD seems to be the only one which has maintained ruby bindings (cf. https://github.com/SpringMT/zstd-ruby).