zstd support for Kafka output

arnitolog commented 3 years ago

Describe the enhancement: It will be really good to have zstd in between Filebeat and Kafka. And it looks like https://github.com/Shopify/sarama/issues/1252 already solved, which unblocks zstd compression implementation in Kafka output. What else needs to be done to have zstd compression as an option for Kafka output?

elasticmachine commented 3 years ago

Pinging @elastic/agent (Team:Agent)

arnitolog commented 2 years ago

Hello, any ETA when zstd compression might be supported?

nimarezainia commented 2 years ago

Hi @arnitolog due to other priorities facing the team right now, we are not considering adding this capability to the beats. We will keep reviewing this. thanks

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

gaby commented 1 year ago

Any updates?

botelastic[bot] commented 6 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

gaby commented 6 months ago

Still valid 👍

gaby commented 4 months ago

Beats already has a dependency that provides support for zstd. Here https://github.com/elastic/beats/blob/main/go.mod#L334C2-L334C31

Dependency: https://github.com/klauspost/compress

elastic / beats

zstd support for Kafka output #24534