confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client

https://github.com/confluentinc/confluent-kafka-dotnet/wiki

Apache License 2.0

68 stars 861 forks source link

Compression Consumer #735

Open ashishbhatia22 opened 5 years ago

ashishbhatia22 commented 5 years ago

Description

We will like to enable compression before sending messages to the bus. I've set the following setting:

compression.codec: "snappy"

Is there a way to find out if this setting is taking effect? Also, can we find out the compression efficiency i.e. message size after the compression is enabled?

mhowlett commented 5 years ago

librdkafka doesn't expose statistics for compression ratio AFAICT. The java producer does. This would be good to add (marked as enhancement). you'd need to work it out yourself using the values that are provided (in stats, by broker metrics & by knowing the uncompressed size of your messages), and I think that would not be trivial. I had a play and don't think it's easy to verify compression is turned on, even with debug messages. if you've set the configuration parameter though, and you didn't get an error message, you can be sure it is being used.

ashishbhatia22 commented 5 years ago

Thanks. I was wondering if a command like this will be helfpul to validate

./kafka-log-dirs --bootstrap-server --topic-list CompressionTopic --describe

mhowlett commented 5 years ago

if you want to get really low level, you could use DumpLogSegments: ./bin/kafka-run-class kafka.tools.DumpLogSegments --deep-iteration --files <log-file-name>