joekiller / logstash-kafka

Kafka plugin for Logstash
Apache License 2.0
206 stars 63 forks source link

Add message key to distribute payloads among distinct partitions #42

Closed pablete closed 9 years ago

pablete commented 9 years ago

Problem: Having nil prevented kafka to send messages to different partitions. (nil will be of no use for the partition function, ie: kafka.producer.DefaultPartitioner)

Zlib::crc32(event) uses 250ns aprox. and 'zlib' is part of the standard library, so no dependencies to load.

Reference: http://programmers.stackexchange.com/questions/49550/which-hashing-algorithm-is-best-for-uniqueness-and-speed