search

wvanbergen / kafka

Load-balancing, resuming Kafka consumer for go, backed by Zookeeper.
MIT License
373 stars 141 forks source link

Some problems when I try to run my consumer in two node. #121

Closed HouLinwei closed 6 years ago

HouLinwei commented 6 years ago

I used to run a consumer( just like examples/main.go) on one node(Node A), but the CPU usage is too high, so I copy my program to another node(Node B). Now I have two consumer(the consume the same topic) with same configuration.

But, here comes the problem. I find the second node(Node B)'s CPU usage is very high just as consumer runs on one node. And the first node(Node A)'s CPU usage is almost zero. In my opinion, the two node will consume the same topic and their CPU usage should be the same. But the result is beyond my expectations. Can you point out is there anything wrong with me?

I will be appreciative. Thank you anyway.

cch123 commented 6 years ago

you need to have more than 1 partition in your topic

HouLinwei commented 6 years ago

@cch123 Thank you for you clue.

Actually I have 8 partitions about this topic, and I just start 8 consumers in Node A. This is the root of my problem. I modify the configuration and it works!

Thank you! And I will close this issue.

Allianzcortex commented 6 years ago

@HouLinwei so start 4 consumers in each node will solve the problem?