search

RuckusWirelessIL / pentaho-kafka-consumer

Apache Kafka consumer step plug-in for Pentaho Kettle
Apache License 2.0
66 stars 40 forks source link

process parition specific data #30

Open leenaKhilnani18 opened 6 years ago

leenaKhilnani18 commented 6 years ago

Is the key field same as the partition number? if not how i can I specify for consumer to read only specific partition data for ApacheKafkaConsumer step(version 1.7)

spektom commented 6 years ago

Key in Kafka used for spreading messages over different partitions.

"In Kafka producer, a partition key can be specified to indicate the destination partition of the message. By default, a hashing-based partitioner is used to determine the partition id given the key, and people can use customized partitioners also." ( https://cwiki.apache.org/confluence/display/KAFKA/FAQ).

You can build your logic of reading from a particular partition based on key value combined with custom partitioner ( https://howtoprogram.xyz/2016/06/04/write-apache-kafka-custom-partitioner/)

On Thu, Jun 21, 2018 at 11:51 PM leenaKhilnani18 notifications@github.com wrote:

Is the key field same as the partition number? if not how i can I specify for consumer to read only specific partition data for ApacheKafkaConsumer step(version 1.7)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RuckusWirelessIL/pentaho-kafka-consumer/issues/30, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpO2VjjCxh4V78V65oSRvoZbgle3lZks5t_AdFgaJpZM4UyvzB .