search

doujiang24 / lua-resty-kafka

Lua kafka client driver for the Openresty based on the cosocket API
BSD 3-Clause "New" or "Revised" License
801 stars 274 forks source link

SASL module failed on specific partitions #122

Closed salon0630 closed 2 years ago

salon0630 commented 2 years ago

Hi, thanks for providing lua-resty-kafka module, which is great module for us sending message from our Openresty to Kafka. Recently, there's a need for us to use Kafka Quota functions, we thought it'd better used along with SASL enabled. We downloaded lua-resty-kafka master branch and having some tests, but weird things happened: the SASL work, yet somehow some of the partitions failed to send messages. E.g. we have 3 partitions, while partition 0 will fail, but partition 1, 2 success. we try to extend the partitions to 9, then we have 0, 3, 6 failed, while other partitions success.

Also we've use kcat command with SASL enabled on Kafka for A/B testing, which work flawlessly. Attached is the error logs from Openresty, Kafka

lua resty kafka error log:

截圖 2022-03-14 下午4 14 23

Kafka error log:

截圖 2022-03-14 下午4 23 55

Please let us know what else information we have to provide, thanks for reading our problem and your works

salon0630 commented 2 years ago

We found the reason which may cause the abnormally

the client.lua file 149, j=1, where Kafka brokers should start from index 0 that's why broker0 always failed

https://github.com/doujiang24/lua-resty-kafka/blob/73204492ceae8210b7dc22a3c34f04da8759ccdf/lib/resty/kafka/client.lua#L149

does this j=1 has specific reason?

doujiang24 commented 2 years ago

oh, I think it's a bug, we'd better use pairs here. Could you please create a PR to fix it? Thanks.

salon0630 commented 2 years ago

PR created. https://github.com/doujiang24/lua-resty-kafka/pull/123