search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
204 stars 3.14k forks source link

why it did not call rebalanceCallback? consumer can't poll message #1781

Closed heketong closed 6 years ago

heketong commented 6 years ago

my consumer can't poll message,so i open the debug log Another thing is when i change the groupid to consume,it will work Compare two situations,i find not call rebalanceCallback, what caused it????????? *** detail logs for not work**** 1524281233.078 RDKAFKA-7-MEMBERID: rdkafka#consumer-1: [thrd:app]: Group "CBE_DSF_VOICE00_20_9520": updating member id "(not-set)" -> "" 1524281233.078 RDKAFKA-7-BRKREASSIGN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" management reassigned from broker (none) to :0/internal 1524281233.078 RDKAFKA-7-CGRPSTATE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changed state init -> wait-broker (v1, join-state init) 1524281233.078 RDKAFKA-7-BRKASSIGN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" management assigned to broker :0/internal 1524281233.079 RDKAFKA-7-CGRPOP: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" received op SUBSCRIBE (v0) in state wait-broker (join state init, v1 vs 0) 1524281233.079 RDKAFKA-7-SUBSCRIBE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": subscribe to new subscription of 1 topics (join state init) 1524281233.079 RDKAFKA-7-UNSUBSCRIBE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": unsubscribe from current unset subscription of 0 topics (leave group=no, join state init, v1) 1524281233.079 RDKAFKA-7-GRPLEADER: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": resetting group leader info: unsubscribe 1524281233.079 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changed join state init -> wait-unassign (v1, state wait-broker) 1524281233.079 RDKAFKA-7-UNASSIGN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": unassign done in state wait-broker (join state wait-unassign): without new assignment: unassign (no previous assignment) 1524281233.079 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changed join state wait-unassign -> init (v1, state wait-broker) 1524281233.079 RDKAFKA-7-CGRPQUERY: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": no broker available for coordinator query: intervaled in state wait-broker 1524281233.088 RDKAFKA-7-LEADER: rdkafka#consumer-1: [thrd:10.64.131.83:9092/bootstrap]: 10.64.131.83:9092/0: Mapped 0 partition(s) to broker 1524281233.088 RDKAFKA-7-LEADER: rdkafka#consumer-1: [thrd:10.64.131.83:9094/bootstrap]: 10.64.131.83:9094/2: Mapped 0 partition(s) to broker 1524281233.088 RDKAFKA-7-SUBSCRIPTION: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": no topics in metadata matched subscription 1524281233.088 RDKAFKA-7-LEADER: rdkafka#consumer-1: [thrd:10.64.131.83:9093/bootstrap]: 10.64.131.83:9093/1: Mapped 0 partition(s) to broker 1524281233.088 RDKAFKA-7-BRKREASSIGN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" management reassigned from broker :0/internal to 10.64.131.83:9094/2 1524281233.088 RDKAFKA-7-BRKUNASSIGN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" management unassigned from broker handle :0/internal 1524281233.088 RDKAFKA-7-BRKASSIGN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" management assigned to broker 10.64.131.83:9094/2 1524281233.088 RDKAFKA-7-CGRPQUERY: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9092/0: Group "CBE_DSF_VOICE00_20_9520": querying for coordinator: intervaled in state wait-broker 1524281233.091 RDKAFKA-7-CGRPCOORD: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9092/0: Group "CBE_DSF_VOICE00_20_9520" coordinator is 10.64.131.83:9094 id 2 1524281233.091 RDKAFKA-7-CGRPCOORD: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changing coordinator -1 -> 2 1524281233.091 RDKAFKA-7-CGRPSTATE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changed state wait-broker -> wait-broker-transport (v1, join-state init) 1524281233.091 RDKAFKA-7-CGRPSTATE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changed state wait-broker-transport -> up (v1, join-state init) 1524281233.091 RDKAFKA-7-JOIN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": join with 0 (1) subscribed topic(s) 1524281233.091 RDKAFKA-7-CGRPMETADATA: rdkafka#consumer-1: [thrd:main]: consumer join: metadata for subscription only available for 0/1 topics (-1ms old) 1524281233.091 RDKAFKA-7-JOIN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": postponing join until up-to-date metadata is available 1524281233.094 RDKAFKA-7-METADATA: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9092/0: 1/1 requested topic(s) seen in metadata 1524281233.094 RDKAFKA-7-SUBSCRIPTION: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": effective subscription list changed from 0 to 1 topic(s): 1524281233.094 RDKAFKA-7-SUBSCRIPTION: rdkafka#consumer-1: [thrd:main]: Topic CBE_DSF_VOICE00_20_9520 with 3 partition(s) 1524281233.094 RDKAFKA-7-REJOIN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": subscription updated from metadata change: rejoining group 1524281233.094 RDKAFKA-7-GRPLEADER: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": resetting group leader info: Group rejoin 1524281233.094 RDKAFKA-7-REJOIN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" rejoining in join-state init without an assignment 1524281233.094 RDKAFKA-7-JOIN: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520": join with 1 (1) subscribed topic(s) 1524281233.094 RDKAFKA-7-CGRPMETADATA: rdkafka#consumer-1: [thrd:main]: consumer join: metadata for subscription is up to date (0ms old) 1524281233.094 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "CBE_DSF_VOICE00_20_9520" changed join state init -> wait-join (v1, state up)

*** detail logs for work **** 1524281199.398 RDKAFKA-7-MEMBERID: rdkafka#consumer-1: [thrd:app]: Group "hechen_test": updating member id "(not-set)" -> "" 1524281199.398 RDKAFKA-7-BRKREASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" management reassigned from broker (none) to :0/internal 1524281199.398 RDKAFKA-7-CGRPSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed state init -> wait-broker (v1, join-state init) 1524281199.398 RDKAFKA-7-BRKASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" management assigned to broker :0/internal 1524281199.399 RDKAFKA-7-CGRPOP: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" received op SUBSCRIBE (v0) in state wait-broker (join state init, v1 vs 0) 1524281199.399 RDKAFKA-7-SUBSCRIBE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": subscribe to new subscription of 1 topics (join state init) 1524281199.399 RDKAFKA-7-UNSUBSCRIBE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": unsubscribe from current unset subscription of 0 topics (leave group=no, join state init, v1) 1524281199.399 RDKAFKA-7-GRPLEADER: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": resetting group leader info: unsubscribe 1524281199.399 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed join state init -> wait-unassign (v1, state wait-broker) 1524281199.399 RDKAFKA-7-UNASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": unassign done in state wait-broker (join state wait-unassign): without new assignment: unassign (no previous assignment) 1524281199.399 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed join state wait-unassign -> init (v1, state wait-broker) 1524281199.399 RDKAFKA-7-CGRPQUERY: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": no broker available for coordinator query: intervaled in state wait-broker 1524281199.414 RDKAFKA-7-LEADER: rdkafka#consumer-1: [thrd:10.64.131.83:9092/bootstrap]: 10.64.131.83:9092/0: Mapped 0 partition(s) to broker 1524281199.414 RDKAFKA-7-LEADER: rdkafka#consumer-1: [thrd:10.64.131.83:9093/bootstrap]: 10.64.131.83:9093/1: Mapped 0 partition(s) to broker 1524281199.414 RDKAFKA-7-LEADER: rdkafka#consumer-1: [thrd:10.64.131.83:9094/bootstrap]: 10.64.131.83:9094/2: Mapped 0 partition(s) to broker 1524281199.414 RDKAFKA-7-SUBSCRIPTION: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": no topics in metadata matched subscription 1524281199.414 RDKAFKA-7-BRKREASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" management reassigned from broker :0/internal to 10.64.131.83:9094/2 1524281199.414 RDKAFKA-7-BRKUNASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" management unassigned from broker handle :0/internal 1524281199.414 RDKAFKA-7-BRKASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" management assigned to broker 10.64.131.83:9094/2 1524281199.414 RDKAFKA-7-CGRPQUERY: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9092/0: Group "hechen_test": querying for coordinator: intervaled in state wait-broker 1524281199.418 RDKAFKA-7-CGRPCOORD: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9092/0: Group "hechen_test" coordinator is 10.64.131.83:9094 id 2 1524281199.418 RDKAFKA-7-CGRPCOORD: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changing coordinator -1 -> 2 1524281199.418 RDKAFKA-7-CGRPSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed state wait-broker -> wait-broker-transport (v1, join-state init) 1524281199.418 RDKAFKA-7-CGRPSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed state wait-broker-transport -> up (v1, join-state init) 1524281199.418 RDKAFKA-7-JOIN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": join with 0 (1) subscribed topic(s) 1524281199.419 RDKAFKA-7-CGRPMETADATA: rdkafka#consumer-1: [thrd:main]: consumer join: metadata for subscription only available for 0/1 topics (-1ms old) 1524281199.419 RDKAFKA-7-JOIN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": postponing join until up-to-date metadata is available 1524281199.423 RDKAFKA-7-METADATA: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9092/0: 1/1 requested topic(s) seen in metadata 1524281199.423 RDKAFKA-7-SUBSCRIPTION: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": effective subscription list changed from 0 to 1 topic(s): 1524281199.423 RDKAFKA-7-SUBSCRIPTION: rdkafka#consumer-1: [thrd:main]: Topic CBE_DSF_VOICE00_20_9520 with 3 partition(s) 1524281199.423 RDKAFKA-7-REJOIN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": subscription updated from metadata change: rejoining group 1524281199.423 RDKAFKA-7-GRPLEADER: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": resetting group leader info: Group rejoin 1524281199.423 RDKAFKA-7-REJOIN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" rejoining in join-state init without an assignment 1524281199.423 RDKAFKA-7-JOIN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": join with 1 (1) subscribed topic(s) 1524281199.423 RDKAFKA-7-CGRPMETADATA: rdkafka#consumer-1: [thrd:main]: consumer join: metadata for subscription is up to date (0ms old) 1524281199.423 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed join state init -> wait-join (v1, state up) 1524281199.429 RDKAFKA-7-JOINGROUP: rdkafka#consumer-1: [thrd:main]: JoinGroup response: GenerationId 339, Protocol range, LeaderId rdkafka-0ef297d5-627e-4cc1-814d-1ed1783ec133 (me), my MemberId rdkafka-0ef297d5-627e-4cc1-814d-1ed1783ec133, 1 members in group: (no error) 1524281199.429 RDKAFKA-7-MEMBERID: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": updating member id "" -> "rdkafka-0ef297d5-627e-4cc1-814d-1ed1783ec133" 1524281199.429 RDKAFKA-7-JOINGROUP: rdkafka#consumer-1: [thrd:main]: Elected leader for group "hechen_test" with 1 member(s) 1524281199.429 RDKAFKA-7-GRPLEADER: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": resetting group leader info: JoinGroup response clean-up 1524281199.429 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed join state wait-join -> wait-metadata (v1, state up) 1524281199.434 RDKAFKA-7-METADATA: rdkafka#consumer-1: [thrd:main]: 10.64.131.83:9094/2: 1/1 requested topic(s) seen in metadata 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" running range assignment for 1 member(s): 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: Member "rdkafka-0ef297d5-627e-4cc1-814d-1ed1783ec133" (me) with 1 subscription(s): 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: CBE_DSF_VOICE00_20_9520 [-1] 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: range: Topic CBE_DSF_VOICE00_20_9520 with 3 partition(s) and 1 subscribing member(s) 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: range: Member "rdkafka-0ef297d5-627e-4cc1-814d-1ed1783ec133": assigned topic CBE_DSF_VOICE00_20_9520 partitions 0..2 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" range assignment for 1 member(s) finished in 0.043ms: 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: Member "rdkafka-0ef297d5-627e-4cc1-814d-1ed1783ec133" (me) assigned 3 partition(s): 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: CBE_DSF_VOICE00_20_9520 [0] 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: CBE_DSF_VOICE00_20_9520 [1] 1524281199.434 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: CBE_DSF_VOICE00_20_9520 [2] 1524281199.434 RDKAFKA-7-ASSIGNOR: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": "range" assignor run for 1 member(s) 1524281199.434 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed join state wait-metadata -> wait-sync (v1, state up) 1524281199.439 RDKAFKA-7-SYNCGROUP: rdkafka#consumer-1: [thrd:main]: SyncGroup response: Success (51 bytes of MemberState data) 1524281199.439 RDKAFKA-7-ASSIGN: rdkafka#consumer-1: [thrd:main]: Group "hechen_test": delegating assign of 3 partition(s) to application rebalance callback on queue rd_kafka_cgrp_new: new assignment 1524281199.439 RDKAFKA-7-CGRPJOINSTATE: rdkafka#consumer-1: [thrd:main]: Group "hechen_test" changed join state wait-sync -> wait-assign-rebalance_cb (v1, state up) rebalanceCallback begin

edenhill commented 6 years ago

Please fill out the issue checklist: https://github.com/edenhill/librdkafka/issues/new