search

joekiller / logstash-kafka

Kafka plugin for Logstash
Apache License 2.0
206 stars 63 forks source link

No broker partitions consumed by consumer thread #16

Closed gideononline closed 10 years ago

gideononline commented 10 years ago

My environment: -- logstash 1.4.0 (installed kafka support using your build instruction) -- elasticsearch 1.1.0 (16 node cluster) -- kafka_0.8.0-2 (3 node cluster) -- zookeeper 3.3.5--1 (3 node quorum running on the kafka cluster)

I have 6 logstash indexers set up to connect to the kafka cluster of 3 nodes as input to logstash.

Three of the logstash indexers are able to connect and retrieve logs. However, other three nodes are getting the following error:

log4j, [2014-04-15T06:04:17.763] WARN: kafka.consumer.ZookeeperConsumerConnector: [logstash_logstash-indexer-i-db49728b-1397541855799-bd574597], No broker partitions consumed by consumer thread logstash_logstash-indexer-i-db49728b-1397541855799-bd574597-0 for topic convertro_logstash

Any help?

regards, Sunny

joekiller commented 10 years ago

How many partitions do you have per Kafka topic? You cannot have more consumer threads than partitions per consumer work group. I'm guessing you have more consumers than partitions on your logging target. On Apr 15, 2014 2:08 AM, "gideononline" notifications@github.com wrote:

My environment: -- logstash 1.4.0 (installed kafka support using your build instruction) -- elasticsearch 1.1.0 (16 node cluster) -- kafka_0.8.0-2 (3 node cluster) -- zookeeper 3.3.5--1 (3 node quorum running on the kafka cluster)

I have 6 logstash indexers set up to connect to the kafka cluster of 3 nodes as input to logstash.

Three of the logstash indexers are able to connect and retrieve logs.

However, other three nodes are getting the following error:

log4j, [2014-04-15T06:04:17.763] WARN: kafka.consumer.ZookeeperConsumerConnector: [logstash_logstash-indexer-i-db49728b-1397541855799-bd574597], No broker partitions consumed by consumer thread logstash_logstash-indexer-i-db49728b-1397541855799-bd574597-0 for topic convertro_logstash

Any help?

regards, Sunny

Reply to this email directly or view it on GitHubhttps://github.com/joekiller/logstash-kafka/issues/16 .

gideononline commented 10 years ago

That was it. I should've looked at the doc more carefully. Thanks again!

bianchaorun commented 10 years ago

I have the same number of partitions and consumers,but only one consumer can consumer the partitions ,others don't work.Why?

joekiller commented 10 years ago

This really should be on the logstash-users email list. You need to include your configurations and any error messages. Earlier in this issue the problem was determined to be a configuration problem. On Sep 16, 2014 11:27 PM, "bianchaorun" notifications@github.com wrote:

I have the same number of partitions and consumers,but only one consumer can consumer the partitions ,others don't work.Why?

— Reply to this email directly or view it on GitHub https://github.com/joekiller/logstash-kafka/issues/16#issuecomment-55844606 .

bianchaorun commented 10 years ago

Thanks for your answer and I'm sorry that it's so long to see and reply for your messages My environment: --kafka_2.10-0.8.1.1(2 nodes cluster) --zookeeper-3.3.4(2 node quorum running on the kafka I have one topic.And it has three partitions.So I use three consumer to consume the partitions. But only one consumer works,others do not anything.When I kill the consumer that works,it will one of them begin to work. the following is the consumer configuration file contents:

zookeeper.connect=51.96.2.85:2181

,127.0.0.1:2182,127.0.0.1:2183

zookeeper.connectiontimeout.ms=1000000

consumer group id

group.id=4 consumer.id=2 client.id=2

consumer timeout

consumer.timeout.ms=5000

auto.commit.enable=true

auto.commit.interval.ms=60000

queued.max.message.chunks=10000

the following is the log for the consumer:

[QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] Stopping leader finder thread [QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] Stopping all fetchers [QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] All connections stopped [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared all relevant queues for this fetcher [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared the data chunks in all the consumer message iterators [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Committing all offsets after clearing the fetcher queues [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Releasing partition ownership [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3 rebalancing the following partitions: ArrayBuffer(0, 1, 2) for topic test with consumers: List(4_2-0, 4_2-1, 4_2-2, 4_3-0, 4_3-1, 4_3-2) [QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-2 for topic test [QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-0 for topic test [QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-1 for topic test [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Updating the cache [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3 selected partitions : [QC] INFO [4_3-leader-finder-thread] kafka.utils.Logging$class.info(68) | [4_3-leader-finder-thread], Starting [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], end rebalancing consumer 4_3 try #0

And thanks again!

joekiller commented 10 years ago

Try not setting the consumer id or client id. I think those setting will interfere with the consumer group. On Sep 17, 2014 10:11 PM, "bianchaorun" notifications@github.com wrote:

Thanks for your answer and I'm sorry that it's so long to see and reply for your messages My environment: --kafka_2.10-0.8.1.1(2 nodes cluster) --zookeeper-3.3.4(2 node quorum running on the kafka I have one topic.And it has three partitions.So I use three consumer to consume the partitions. But only one consumer works,others do not anything.When I kill the consumer that works,it will one of them begin to work. the following is the consumer configuration file contents:

zookeeper.connect=51.96.2.85:2181

,127.0.0.1:2182,127.0.0.1:2183

timeout in ms for connecting to zookeeper

zookeeper.connectiontimeout.ms=1000000

consumer group id

group.id=4 consumer.id=2 client.id=2

consumer timeout

consumer.timeout.ms=5000

auto.commit.enable=true

auto.commit.interval.ms=60000

queued.max.message.chunks=10000

the following is the log for the consumer:

[QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] Stopping leader finder thread [QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] Stopping all fetchers [QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] All connections stopped [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared all relevant queues for this fetcher [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared the data chunks in all the consumer message iterators [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Committing all offsets after clearing the fetcher queues [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Releasing partition ownership [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3 rebalancing the following partitions: ArrayBuffer(0, 1, 2) for topic test with consumers: List(4_2-0, 4_2-1, 4_2-2, 4_3-0, 4_3-1, 4_3-2) [QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-2 for topic test [QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-0 for topic test [QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-1 for topic test [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Updating the cache [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3 selected partitions : [QC] INFO [4_3-leader-finder-thread] kafka.utils.Logging$class.info(68) | [4_3-leader-finder-thread], Starting [QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], end rebalancing consumer 4_3 try #0

And thanks again!

— Reply to this email directly or view it on GitHub https://github.com/joekiller/logstash-kafka/issues/16#issuecomment-55986900 .