search

Azure / azure-event-hubs-for-kafka

Azure Event Hubs for Apache Kafka Ecosystems
https://docs.microsoft.com/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Other
231 stars 212 forks source link

compression support for zstd #39

Open cforce opened 5 years ago

cforce commented 5 years ago

The documentation is for me quit unclear, it says "If compressed data is necessary, compressing your data before sending it to the brokers and decompressing after receiving it is a valid workaround. The message body is just a byte array to the service, so client-side compression/decompression will not cause any issues." Does this mean that i can use the kafka feature that does compression on client side (instead of passing "--config compression.type=gzip" at topic creation) and on producer side so the broker therefore does not need to compress by itself. Or does it mean kafka compression even in this approach will be blocked by the server. In that case this is a feature request. For WAN connections where IOT field gatways or on premise apps produce msgs to the iot hub , it's it mandatory to use compression (high volume) because of costs of reliable VPNs.

ShubhaVijayasarathy commented 5 years ago

If I am understanding this request correctly, it is log compression that you are asking?

cforce commented 5 years ago

I mean compression of the messages going over the brokered tcp connections -not something on disk

ShubhaVijayasarathy commented 5 years ago

yes today we do not compress the events. This needs to be in your clients. Let me take this as a user ask and we have more customers asking for the same, we will prioritize to implement it.

cforce commented 5 years ago

From my understanding kafka protocol clients already can do that on client side as it is also possible by config per topic on broker side while having producer->broker uncompressed. I am mainly asking for the support of (de)compression done on client side, but using the kafka protocol .

image

Producers(>=2.1): compression.type=zstd Consumers (<2.1): UNSUPPORTED_COMPRESSION_TYPE (error) Consumers(>=2.1): Working Brokers v 2.1 (Kafka).

Link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression

david-esteves-sh commented 1 year ago

Is there still no plan to support this? Using open-telemetry collector for example and the kafka-exporter, the compression options do not work when using eventhubs for kafka. There's no way to compress the body data in this flow, so it would be great if EventHubs could support this!

kinglite commented 7 months ago

Is this related to the options one gets in Azure Portal when, for example, setting up an input of type Event Hub in a Stream Analytics job and need to define the type of compression of events? Because there I also miss the option to use zstd. (Or at least give an option that would allow to read it raw and decompress it via user defined function)