search

kafkaex / kafka_ex

Kafka client library for Elixir
MIT License
595 stars 162 forks source link

Unable to use consumer groups #210

Closed lbrito1 closed 7 years ago

lbrito1 commented 7 years ago

I've been using the latest (0.65) version successfully with a dockerized Kafka 0.10.2.1. For testing purposes, I have a single topic with one partition and one replica. I have set consumer_group: "elixir_default" on my config.exs.

Fetching works fine. However, when I try to use the consumer_group option in a worker (with any value - be it the default group or not), I get a consumer_coordinator_not_available error:

KafkaEx.create_worker(:xyz, [consumer_group: "abcd"])
KafkaEx.fetch(@topic, 0, offset: 0, worker_name: :xyz)
# 18:05:59.689 [error] Fetching consumer_group abcd metadata failed with
# error_code :consumer_coordinator_not_available

Zookeeper spills the following error:

zookeeper    | 2017-06-15 18:05:59,696 [myid:] - INFO  [ProcessThread(sid:0 
cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when
processing sessionid:0x15cac2448f80000 type:create cxid:0x163 zxid:0x51e 
txntype:-1 reqpath:n/a Error Path:/consumers/abcd/offsets Error:KeeperErrorCode
 = NoNode for /consumers/abcd/offsets

I suspect this might have to do with other issues, but I don't have enough knowledge on the subject yet to say to. Or maybe I'm missing some configuration

dantswain commented 7 years ago

@lbrito1 Have you tried publishing to the topic before consuming from it? I have seen similar issues but they are usually transient (i.e., works on a second run, or a restart of everything).

lbrito1 commented 7 years ago

@dantswain The topic had a few messages prior to this test. I'll try restarts and will report back with the results.

lbrito1 commented 7 years ago

I can confirm that restarting the services (Kafka, ZK and Elixir docker containers) solved the issue in my case. Thanks for the help @dantswain !

dantswain commented 7 years ago

:+1: Great to hear! I hope we can get rid of this error in general, but I'm glad you were able to move forward.

dopatraman commented 5 years ago

This problem has resurfaced for me with a slightly different face.

I'm trying to produce to a group kafka_ex11 and getting this error:

Fetching consumer_group kafka_ex11 metadata failed with error_code :group_authorization_failed

This is my producer:

KafkaEx.create_worker(name, [consumer_group: "kafka_ex11"])

and consumer:

KafkaEx.create_worker(name, [consumer_group: "kafka_ex11"])

What am i doing wrong?

joshuawscott commented 5 years ago

You have to produce to a topic, not a consumer group.

joshuawscott commented 5 years ago

Looking at this further, it appears you have ACLs set up to control access to the consumer group. KafkaEx unfortunately doesn't support ACLs at this point, although PRs are always welcome!