search

mailgun / kafka-pixy

gRPC/REST proxy for Kafka
Apache License 2.0
768 stars 119 forks source link

Empty consumer group name should not be allowed #110

Closed gyan777 closed 7 years ago

gyan777 commented 7 years ago

I have looked around quite a bit but have not been able to find the reason or fix for the following Zookeeper errors that I see printed on kafka-Pixy output console when the consumer connects. I have tried both C++ and Python gRPC stubs but the end result is the same.

Jun 23 19:56:42.894 INFO [sarama] Connected to broker at localhost:9092 (registered as #0) Jun 23 19:56:47.054 INFO </default[0]/cons[0]/G:[0]> started Jun 23 19:56:47.054 INFO </default[0]/cons[0]/G:[0]/dispatcher[0]> started Jun 23 19:56:47.054 INFO </default[0]/cons[0]/G:[0]/member[0]> started Jun 23 19:56:47.054 INFO </default[0]/cons[0]/G:[0]/T:myMsgQueue[0]> started Jun 23 19:56:47.054 INFO </default[0]/cons[0]/G:[0]/manager[0]> started Jun 23 19:56:47.054 INFO </default[0]/cons[0]/G:[0]/msg_stream_f[0]/mapper[0]> started Jun 23 19:56:47.059 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 23 19:56:47.570 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error)

I start Kafka in a container as follows:

docker pull spotify/kafka docker run --rm -p 2181:2181 -p 9092:9092 --env \ ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 \ --name kafka -h kafka spotify/kafka

horkhe commented 7 years ago

I just tried using Kafka-Pixy with the spotify/kafka image and it worked for me. I used curl to test the setup rather than a gRPC generated client, but that does not make any difference. So could you please write a bit more about your setup:

gyan777 commented 7 years ago

Attaching the full log below:

$:~/kafka-pixy-v0.13.0-linux-amd64> ./kafka-pixy --config config.yaml Jun 26 16:22:28.539 INFO Starting with config: &{GRPCAddr:0.0.0.0:19091 TCPAddr:0.0.0.0:19092 UnixAddr: Proxies:map[default:0xc4201f8000] DefaultCluster:default} Jun 26 16:22:28.539 INFO [sarama] Initializing new client Jun 26 16:22:28.539 INFO [sarama] client/metadata fetching metadata for all topics from broker localhost:9092 Jun 26 16:22:28.540 INFO [sarama] Connected to broker at localhost:9092 (unregistered) Jun 26 16:22:28.541 INFO [sarama] client/brokers registered new broker #0 at localhost:9092 Jun 26 16:22:28.541 INFO [sarama] Successfully initialized new client Jun 26 16:22:28.541 INFO [sarama] Initializing new client Jun 26 16:22:28.541 INFO [sarama] client/metadata fetching metadata for all topics from broker localhost:9092 Jun 26 16:22:28.541 INFO </default[0]/offset_mgr_f[0]/mapper[0]> started Jun 26 16:22:28.541 INFO [sarama] Connected to broker at localhost:9092 (unregistered) Jun 26 16:22:28.542 INFO [sarama] client/brokers registered new broker #0 at localhost:9092 Jun 26 16:22:28.542 INFO [sarama] Successfully initialized new client Jun 26 16:22:28.542 INFO [sarama] Initializing new client Jun 26 16:22:28.542 INFO </default[0]/prod[0]/merger[0]> started Jun 26 16:22:28.542 INFO [sarama] client/metadata fetching metadata for all topics from broker localhost:9092 Jun 26 16:22:28.542 INFO </default[0]/prod[0]/dispatcher[0]> started Jun 26 16:22:28.542 INFO [sarama] Connected to broker at localhost:9092 (unregistered) Jun 26 16:22:28.543 INFO [sarama] client/brokers registered new broker #0 at localhost:9092 Jun 26 16:22:28.543 INFO [sarama] Successfully initialized new client Jun 26 16:22:28.544 INFO </default[0]/cons[0]/dispatcher[0]> started Jun 26 16:22:28.544 INFO [zk] Connected to 127.0.0.1:2181 Jun 26 16:22:28.544 INFO </service[0]> started Jun 26 16:22:28.544 INFO </http://0.0.0.0:19092[0]> started Jun 26 16:22:28.545 INFO </grpc://0.0.0.0:19091[0]> started Jun 26 16:22:28.549 INFO [zk] Authenticated: id=98188657940627464, timeout=15000 Jun 26 16:22:28.549 INFO [zk] Re-submitting 0 credentials after reconnect Jun 26 16:23:13.176 INFO [sarama] producer/broker/0 starting up Jun 26 16:23:13.176 INFO [sarama] producer/broker/0 state change to [open] on myMsgQueue/0 Jun 26 16:23:13.176 INFO [sarama] Connected to broker at localhost:9092 (registered as #0) Jun 26 16:23:57.432 INFO </default[0]/cons[0]/G:[0]> started Jun 26 16:23:57.432 INFO </default[0]/cons[0]/G:[0]/dispatcher[0]> started Jun 26 16:23:57.432 INFO </default[0]/cons[0]/G:[0]/T:myMsgQueue[0]> started Jun 26 16:23:57.432 INFO </default[0]/cons[0]/G:[0]/member[0]> started Jun 26 16:23:57.432 INFO </default[0]/cons[0]/G:[0]/manager[0]> started Jun 26 16:23:57.432 INFO </default[0]/cons[0]/G:[0]/msg_stream_f[0]/mapper[0]> started Jun 26 16:23:57.441 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:23:57.949 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:23:58.459 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:23:58.969 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:23:59.476 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:23:59.986 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:00.496 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:01.005 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:01.515 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:02.026 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:02.536 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:03.045 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:03.555 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:04.065 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:04.575 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:05.085 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:05.596 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:06.106 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:06.616 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:07.126 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:07.636 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:08.146 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:08.656 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:09.166 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:09.676 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:10.186 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:10.697 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:11.207 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:11.717 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:12.227 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:12.737 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error) Jun 26 16:24:13.247 ERROR </default[0]/cons[0]/G:[0]/member[0]> failed to create a group znode: err=(zk: unknown error)

gyan777 commented 7 years ago

My config.yaml is just a copy of the default.yaml that is provided in the distribution. The commandline and the version is the first line the cut-n-paste of the log. Both C++ and python clients lead to the same error.

horkhe commented 7 years ago

The reason it is failing for you is that ConsNAckRq.group must not be empty, you have to pass some group name. I will use this issue to make Kafka-Pixy return an error in such cases.

gyan777 commented 7 years ago

Putting in a group name works. Thanks.