confluentinc / kafka-rest

Confluent REST Proxy for Kafka
https://docs.confluent.io/current/kafka-rest/docs/index.html
Other
38 stars 643 forks source link

Configure Field in message to be used as Message Key in Kafka partitioning #1190

Open brshravan-tech opened 1 year ago

brshravan-tech commented 1 year ago

Hi Team,

Here is my use case. I have a server which calls Rest API calls (using fluent-bit) to Kafka Rest server which internally pushes the messages to Azure Eventhub. I am looking at possibility of defining field in the record which can be passed to Azure Eventhub to be considered as Message Key which internally will be used for deciding the partition. Basically, what I am looking at is, defining some field in the record as partition key so that all the records having the same value in the mentioned field goes to same partition. Do we have any such mechanism or flexibility available in Kafka Rest server? FYI, we are using Kafka Rest Server as Kubernetes services (from confluentinc/cp-kafka-rest). So, I am looking at defining or configuring this through the environment variables in Kubernetes Yaml file or any other mechanisms.

Thanks in advance.