doujiang24 / lua-resty-kafka

Lua kafka client driver for the Openresty based on the cosocket API
BSD 3-Clause "New" or "Revised" License
801 stars 274 forks source link

Add handler for successful send to Kafka #30

Closed gemelen closed 7 years ago

gemelen commented 8 years ago

Goal of proposed change:

ability to control successful sending to Kafka, which is vital for monitoring of complex pipelines. E.g. metric reporting placed in error and success handles to log number of published and failed messages to monitoring or logging facility.

What was changed:

  1. in case of successful sending, handling of this case is added. We call successful handler if any or write to log with Nginx logging API.
  2. additional case triggers error handler. This occurs when we have no partitions metadata (brokers are down), but still have non empty message queue.
  3. shuffle function for tables
  4. broker list shuffled on each producer:new call
doujiang24 commented 7 years ago

@gemelen Thanks you very much :) And really sorry for the delay on my side :< I think the deepcopy for every sendbuffer may bring performance problem even when no success_handler :) May be you can simply call the success_handler in _send, and we can get more useful information there, like offset:) https://github.com/doujiang24/lua-resty-kafka/blob/master/lib/resty/kafka/producer.lua#L154

gemelen commented 7 years ago

Thanks for reply and for your suggestions regarding this code improvement. Idea to avoid additional buffer is great, even if we already do not see much impact from this chage in production. I'll add implementation in short time

gemelen commented 7 years ago

Changelist is updated, suggestions on change applied.