search

doujiang24 / lua-resty-kafka

Lua kafka client driver for the Openresty based on the cosocket API
BSD 3-Clause "New" or "Revised" License
804 stars 275 forks source link

Manually select Kafka partition #3

Closed carminexx closed 8 years ago

carminexx commented 9 years ago

Hi, Is it possibile to manually select the kafka partition, when sending messages? I noticed that in the current implementation all messages are sent to partition 1, is it correct?

doujiang24 commented 9 years ago

It's recommended to use bufferproducer, this will manually select the kafka partition (just simple round). And bufferproducer means higher performance.

carminexx commented 9 years ago

Thanks for your reply; but I noticed that the partition is automatically selected even in buffer producer. For "manually" I mean that I can choose exactly to which partition send the message.

doujiang24 commented 9 years ago

Sorry for the misunderstood, but I not exactly understand your idea. I don't think It's a good idea to specifie a partition in send api. Do you mean specifie a key in send api, and a partitioner function in producer config. The partitioner choose partition according to the key?

I have look into some kafka client in other language(like go) and the kafka protocol, seems the second one is better. And I don't see much sense to provide this in openresty world, could you show me?

carminexx commented 9 years ago

I mean something in order to have, for example, messages relatives to same user in the same Kafka partition. An example: I have a web application with user authentication and user tracking via cookies. I want that messages relatives to an user are sent to a specified partition, and the partition will be chosen via LUA after an analysis of a user tracking-code, stored in a HTTP cookie. Ex: user 1 => tracking code 2 => all messages sent to partition 1 user 2 => tracking code 3 => all messages sent to partition 2 user 3 => tracking code 8 => all messages sent to partition 1 (in this case, via LUA code, I sent messages for users that have an even tracking code to partition 1; whereas users that have an odd tracking code have messages sent to partition 2).

doujiang24 commented 9 years ago

Thanks, I got you. I will implement a new send api in the two days, or maybe done tomorrow.

doujiang24 commented 9 years ago

Sorry, later for the $ work, may be finished in this weekend.

doujiang24 commented 9 years ago

Sorry for the later again. Finally I finished a implement in the master branch now. Any feedback welcome.

doujiang24 commented 8 years ago

Consider it resolved.