hoobean1996 commented 3 years ago

Description

How to reproduce

Sorry currently not find the related clues. can you please give me some ideas about what could be wrong with our case? Another background is our Kafka Broker for it has high CPU usage.

Thank you!!

Checklist

Please provide the following information:

[ ] confluent-kafka-go and librdkafka version (LibraryVersion()): kafka.LibraryVersion(): 17170943 1.6.1
[ ] Apache Kafka broker version: confluent-kafka-2.11
[ ] Client configuration: ConfigMap{...} "bootstrap.servers": strings.Join(brokers, ","), "enable.auto.commit": false, "group.id": groupID, "auto.offset.reset": "earliest", "client.id": uuid.New().String(), "session.timeout.ms": 1000 * 30, "debug": "cgrp,consumer",
[ ] Operating system: Ubuntu 16.04
[ ] Provide client logs (with "debug": ".." as necessary)

// Logs without details information
// All stuck consume have this kind of logs. 
%4|1618057306.840|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Broker: Not coordinator): revoking assignment and rejoining group
%4|1618221377.882|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10002 ms without a successful response from the group coordinator (broker 3, last error was Broker: Not coordinator): revoking assignment and rejoining group
%4|1618320956.115|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group
%4|1619089508.949|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10261 ms without a successful response from the group coordinator (broker 3, last error was Broker: Not coordinator): revoking assignment and rejoining group
%4|1619213305.354|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10170 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group
%4|1619395406.634|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1619400397.248|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group
%4|1619506734.319|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10002 ms without a successful response from the group coordinator (broker -1, last error was Broker: Not coordinator): revoking assignment and rejoining group
%4|1619531042.957|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10012 ms without a successful response from the group coordinator (broker 2, last error was Broker: Not coordinator): revoking assignment and rejoining group
%4|1619548451.781|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group
%4|1619594346.081|SESSTMOUT|2032410a-6e7f-48bc-97dd-e5baf9d764ce#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10119 ms without a successful response from the group coordinator (broker 2, last error was Broker: Not coordinator): revoking assignment and rejoining group

// Logs previously, here the session time is 10 seconds by default. 
// After we found the timeout, then I try to increase the seesion time, and enable cgrp, and consumer.
%4|1618906283.934|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10268 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1618928326.994|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10481 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1618930921.484|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10008 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1618941303.647|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10293 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1618971141.824|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10263 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1619023533.125|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group
%4|1619119042.827|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1619129873.247|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10163 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group
%4|1619211958.215|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10091 ms without a successful response from the group coordinator (broker 2, last error was Local: Broker transport failure): revoking assignment and rejoining group
%4|1619213305.199|SESSTMOUT|d0f13160-2ad1-4437-929c-fd79c1b73095#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 10000 ms without a successful response from the group coordinator (broker 2, last error was Success): revoking assignment and rejoining group

// Logs enabled with cgrp, consumer, we updated the session time to longer.
%7|1619405978.698|CGRPQUERY|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local": querying for coordinator: Broker: Not coordinator
%7|1619405978.699|CGRPCOORD|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local" coordinator is 10.143.182.205:9092 id 0
%7|1619405980.697|JOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local": join with 1 subscribed topic(s)
%7|1619405980.697|CGRPMETADATA|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: consumer join: metadata for subscription is up to date (213189ms old)
%7|1619405980.697|JOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Joining group "kale_scheduler_local" with 1 subscribed topic(s)
%7|1619405980.697|CGRPJOINSTATE|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" changed join state init -> wait-join (state up)
%7|1619405980.698|JOINGROUP|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: JoinGroup response: GenerationId -1, Protocol , LeaderId , my MemberId , member metadata count 0: Broker: Not coordinator
%7|1619405980.698|REJOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local": Rejoining group without an assignment: JoinGroup error: Broker: Not coordinator
%7|1619405980.698|CGRPJOINSTATE|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" changed join state wait-join -> init (state up)
%7|1619405980.698|CGRPOP|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" received op COORD_QUERY in state up (join-state init)
%7|1619405980.698|CGRPQUERY|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local": querying for coordinator: Broker: Not coordinator
%7|1619405980.699|CGRPCOORD|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local" coordinator is 10.143.182.205:9092 id 0
%7|1619405982.697|JOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local": join with 1 subscribed topic(s)
%7|1619405982.697|CGRPMETADATA|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: consumer join: metadata for subscription is up to date (215189ms old)
%7|1619405982.697|JOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Joining group "kale_scheduler_local" with 1 subscribed topic(s)
%7|1619405982.697|CGRPJOINSTATE|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" changed join state init -> wait-join (state up)
%7|1619405982.698|JOINGROUP|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: JoinGroup response: GenerationId -1, Protocol , LeaderId , my MemberId , member metadata count 0: Broker: Not coordinator
%7|1619405982.698|REJOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local": Rejoining group without an assignment: JoinGroup error: Broker: Not coordinator
%7|1619405982.698|CGRPJOINSTATE|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" changed join state wait-join -> init (state up)
%7|1619405982.698|CGRPOP|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" received op COORD_QUERY in state up (join-state init)
%7|1619405982.698|CGRPQUERY|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local": querying for coordinator: Broker: Not coordinator
%7|1619405982.699|CGRPCOORD|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local" coordinator is 10.143.182.205:9092 id 0
%7|1619405984.697|JOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local": join with 1 subscribed topic(s)
%7|1619405984.697|CGRPMETADATA|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: consumer join: metadata for subscription is up to date (217189ms old)
%7|1619405984.697|JOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Joining group "kale_scheduler_local" with 1 subscribed topic(s)
%7|1619405984.697|CGRPJOINSTATE|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" changed join state init -> wait-join (state up)
%7|1619405984.698|JOINGROUP|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: JoinGroup response: GenerationId -1, Protocol , LeaderId , my MemberId , member metadata count 0: Broker: Not coordinator
%7|1619405984.698|REJOIN|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local": Rejoining group without an assignment: JoinGroup error: Broker: Not coordinator
%7|1619405984.698|CGRPJOINSTATE|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" changed join state wait-join -> init (state up)
%7|1619405984.698|CGRPOP|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: Group "kale_scheduler_local" received op COORD_QUERY in state up (join-state init)
%7|1619405984.698|CGRPQUERY|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local": querying for coordinator: Broker: Not coordinator
%7|1619405984.699|CGRPCOORD|2122b260-99fe-4e98-8d7f-5b6e174612bd#consumer-2| [thrd:main]: 10.143.182.205:9092/0: Group "kale_scheduler_local" coordinator is 10.143.182.205:9092 id 0

[ ] Provide broker log excerpts
[ ] Critical issue we don't stably reproduce this issue yet, but I set the cgrp,consumer debug logs. Got some logs.

When the consumer revoked the partition assignment, it would not get new partition assignment with stuck in this phase.

When the issue happened, it would stuck here, and could not get any new message any more.

hoobean1996 commented 3 years ago

Kafka Broker Logs: We found when the issue happened, it always has some key words like Out of sync.

Node 0:

2021-04-23 08:33:00,523 INFO [GroupCoordinator 0]: Unloading group metadata for kale_scheduler_local with generation 5051 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:29,454 INFO [GroupCoordinator 0]: Loading group metadata for kale_scheduler_local with generation 5052 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:30,143 INFO [GroupCoordinator 0]: Preparing to rebalance group kale_scheduler_local in state PreparingRebalance with old generation 5052 (__consumer_offsets-47) (reason: Adding new member 33f90220-ab5e-4084-a21b-57e9d3f74f58-23e236b7-21c4-4e57-bdab-8d211dfd0d19 with group instanceid None) (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member d75ae538-e997-488a-a27a-8ea716f6672e-60c4da13-093a-43d2-ae35-a5d77db16b62 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 4f481c4c-6238-4463-843e-4cbdc29b6b4f-0a467592-5cf1-4dbf-a781-7b810e6159bd in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member a3a65772-4aa5-48c3-92af-71584429d0e6-da6f8ac3-10e0-43b6-b999-4a72c6b1633e in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 5be974fc-c1cc-4eb7-9295-f4609131edc4-e20f1c11-3f64-4a34-b416-598f9accdc58 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 69efdfe4-53d8-4657-902a-9699fd791cde-621d4c9c-2508-4bf7-af35-53b58c5b2ccb in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 5b17da2d-bbb6-495a-af86-b45752d1e420-3ac41933-c993-48e1-ac8c-c2f8cbf654c9 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member e2d17b79-0409-4ef6-aeca-5415b7b2fb9b-a6e50f5f-99f5-4bd9-adf6-95288c246f09 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member a202d7f6-7a86-43de-ae0b-22644681dc88-2d9ac78c-5b40-477c-8b94-906115bfc6aa in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 2122b260-99fe-4e98-8d7f-5b6e174612bd-a82545b6-014d-4dd1-9e01-77d7fb5512a8 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 69e85c6f-486b-4123-ab7c-7e3a7eb0e9b5-819e35cf-8b4b-4ebe-aa7a-b3be06cc18b3 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 115438c8-cd5f-42f6-b31a-fff44ca46135-e87504c2-579a-4138-a65c-2d117c67edff in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member e2f5e46d-4280-49b4-9c9d-fd95002adfb8-a35bacca-752b-43e4-a122-c54ad6a1ff66 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 851d9d04-7590-4fa1-9c5f-623b717df843-9416eed1-061c-488c-93bd-20b57099da0e in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 5ef55a3c-d61f-45fd-b62e-5e2269cc9eff-184f3e06-daaa-4db2-8190-46357526078d in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 6ae02ce9-62b7-4cee-8d92-6c4e098d5ea6-823754a4-5410-40fe-b098-1460392ca9b0 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 02f6a1fe-de08-48f8-a595-20d63c3ba27c-28d295a5-922f-4944-9a98-aac8b0d8293b in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 7b534150-0b2f-4cfe-a160-e24e8cc282f1-43a5492d-66b5-46e8-9024-c7400902b776 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 33f90220-ab5e-4084-a21b-57e9d3f74f58-7796c9a4-32a5-4696-a887-3a8e322a6106 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,454 INFO [GroupCoordinator 0]: Member 13befb36-fcd6-4d4a-ab4d-a54ad12fec6a-bbc36c02-e03a-4b1a-9f70-23641002b938 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,455 INFO [GroupCoordinator 0]: Member ddab1744-b277-4663-aa1b-79f4f1343ee2-9766b6e3-4e58-49d1-95e8-9b960a203ade in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:59,455 INFO [GroupCoordinator 0]: Member a15f348b-9f98-42f0-a9d1-a019a65791d1-e095da6a-f6ee-43d0-977a-b33973fea4b4 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:37:00,255 INFO [GroupCoordinator 0]: Member ecb2f5ce-b4fe-4a02-89df-a94af19f452b-5ddab973-4c72-4b4f-b621-12129be87f8b in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:37:00,269 INFO [GroupCoordinator 0]: Member d06b7575-ddc9-47f9-9203-5ef6f14c4ca1-2fe34e26-20f2-4f44-bc7c-fafac747e5ad in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:37:00,271 INFO [GroupCoordinator 0]: Stabilized group kale_scheduler_local generation 5053 (__consumer_offsets-47) (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:37:00,278 INFO [GroupCoordinator 0]: Assignment received from leader for group kale_scheduler_local for generation 5053 (kafka.coordinator.group.GroupCoordinator)

Node 1:

2021-04-22 01:45:31,127 INFO [GroupCoordinator 1]: Stabilized group kale_scheduler_local generation 5019 (__consumer_offsets-47) (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:31,147 INFO [GroupCoordinator 1]: Assignment received from leader for group kale_scheduler_local for generation 5019 (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:31,241 INFO [GroupCoordinator 1]: Preparing to rebalance group kale_scheduler_local in state PreparingRebalance with old generation 5019 (__consumer_offsets-47) (reason: Adding new member 29e24eeb-dd6c-43ed-90e2-9264326e4072-79cc225b-0b3c-446b-90cb-47761c53ded6 with group instanceid None) (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:32,353 INFO [GroupCoordinator 1]: Stabilized group kale_scheduler_local generation 5020 (__consumer_offsets-47) (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:32,428 INFO [GroupCoordinator 1]: Assignment received from leader for group kale_scheduler_local for generation 5020 (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:32,601 INFO [GroupCoordinator 1]: Preparing to rebalance group kale_scheduler_local in state PreparingRebalance with old generation 5020 (__consumer_offsets-47) (reason: Adding new member 4e3a85e6-0f55-4094-b196-481c408a11ce-0ac581d0-3671-4534-82f9-a205851ca2ed with group instanceid None) (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:33,356 INFO [GroupCoordinator 1]: Stabilized group kale_scheduler_local generation 5021 (__consumer_offsets-47) (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:33,365 INFO [GroupCoordinator 1]: Assignment received from leader for group kale_scheduler_local for generation 5021 (kafka.coordinator.group.GroupCoordinator)
2021-04-22 01:45:54,740 INFO [GroupCoordinator 1]: Unloading group metadata for kale_scheduler_local with generation 5021 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 07:14:36,865 INFO [GroupCoordinator 1]: Loading group metadata for kale_scheduler_local with generation 5037 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 07:15:06,867 INFO [GroupCoordinator 1]: Member 95b8df8b-97ba-44e7-a4e1-17d2157f7e15-8a47e391-be26-4975-9d0d-b686164e5cba in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 07:15:06,867 INFO [GroupCoordinator 1]: Preparing to rebalance group kale_scheduler_local in state PreparingRebalance with old generation 5037 (__consumer_offsets-47) (reason: removing member 95b8df8b-97ba-44e7-a4e1-17d2157f7e15-8a47e391-be26-4975-9d0d-b686164e5cba on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
2021-04-23 07:15:08,863 INFO [GroupCoordinator 1]: Stabilized group kale_scheduler_local generation 5038 (__consumer_offsets-47) (kafka.coordinator.group.GroupCoordinator)
2021-04-23 07:15:08,870 INFO [GroupCoordinator 1]: Assignment received from leader for group kale_scheduler_local for generation 5038 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 07:15:57,000 INFO [GroupCoordinator 1]: Unloading group metadata for kale_scheduler_local with generation 5038 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:08,128 INFO [GroupCoordinator 1]: Loading group metadata for kale_scheduler_local with generation 5051 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:26,093 INFO [GroupCoordinator 1]: Preparing to rebalance group kale_scheduler_local in state PreparingRebalance with old generation 5051 (__consumer_offsets-47) (reason: Adding new member 866cf202-f57f-417e-a3ac-65d9d1c2ba81-42282b52-c9d2-4e39-bb98-e09b423597f6 with group instanceid None) (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:38,157 INFO [GroupCoordinator 1]: Member d2631866-718c-4697-b7cf-86e009d5b5c7-a39f1eaf-3457-4416-845f-ff74c76c48ce in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:38,157 INFO [GroupCoordinator 1]: Member dee8f3d6-e1f1-4f3f-8a2d-86f650f3def7-4bb8ab12-1992-463c-a7be-718034abe583 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:51,188 INFO [GroupCoordinator 1]: Member 866cf202-f57f-417e-a3ac-65d9d1c2ba81-043cf239-ea5a-4094-8d54-f6e3ee248f65 in group kale_scheduler_local has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:51,235 INFO [GroupCoordinator 1]: Stabilized group kale_scheduler_local generation 5052 (__consumer_offsets-47) (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:33:51,244 INFO [GroupCoordinator 1]: Assignment received from leader for group kale_scheduler_local for generation 5052 (kafka.coordinator.group.GroupCoordinator)
2021-04-23 08:36:00,616 INFO [GroupCoordinator 1]: Unloading group metadata for kale_scheduler_local with generation 5052 (kafka.coordinator.group.GroupCoordinator)

Node 2


2021-04-23 08:33:01,125 INFO [Log partition=kale.marketplace.return.return_noti.de-12, dir=/kafka-data/kafka] Truncating to 1429 has no effect as the largest offset in the log is 1428 (kafka.log.Log)
2021-04-23 08:33:01,125 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,126 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,126 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,148 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-1 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,148 INFO [Log partition=kale.marketplace.return.return_noti.de-3, dir=/kafka-data/kafka] Truncating to 1448 has no effect as the largest offset in the log is 1447 (kafka.log.Log)
2021-04-23 08:33:01,148 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-4 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,149 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-15 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,150 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-10 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,150 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-13 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,151 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-7 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:01,151 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-9 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:04,967 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:04,968 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:04,969 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:04,969 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:04,970 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:04,970 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,632 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-1 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,776 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-4 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,777 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-15 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,778 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-10 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,778 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-13 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,779 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-7 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:06,779 INFO [Log partition=kale.marketplace.return.return_noti.de-9, dir=/kafka-data/kafka] Truncating to 1473 has no effect as the largest offset in the log is 1472 (kafka.log.Log)
2021-04-23 08:33:07,011 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:07,012 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:07,013 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:07,014 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:07,014 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:07,014 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:09,439 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:09,440 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:09,441 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:09,441 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:09,441 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:09,442 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:10,355 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-1 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:10,356 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-4 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:10,356 INFO [Log partition=kale.marketplace.return.return_noti.de-15, dir=/kafka-data/kafka] Truncating to 1496 has no effect as the largest offset in the log is 1495 (kafka.log.Log)
2021-04-23 08:33:10,357 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-10 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:10,357 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-13 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:10,358 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-7 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:11,341 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:11,342 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:11,344 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:11,344 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:11,344 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:11,345 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:13,056 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-1 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:13,056 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-4 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:13,057 INFO [Log partition=kale.marketplace.return.return_noti.de-10, dir=/kafka-data/kafka] Truncating to 1525 has no effect as the largest offset in the log is 1524 (kafka.log.Log)
2021-04-23 08:33:13,057 INFO [Log partition=kale.marketplace.return.return_noti.de-13, dir=/kafka-data/kafka] Truncating to 1443 has no effect as the largest offset in the log is 1442 (kafka.log.Log)
2021-04-23 08:33:13,057 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-7 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:14,044 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:14,044 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:14,045 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:14,046 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:14,046 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:14,046 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:15,681 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-1 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:15,681 INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-4 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:15,682 INFO [Log partition=kale.marketplace.return.return_noti.de-7, dir=/kafka-data/kafka] Truncating to 1442 has no effect as the largest offset in the log is 1441 (kafka.log.Log)
2021-04-23 08:33:17,095 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:17,095 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:17,096 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:17,097 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:17,097 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:17,097 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:17,183 INFO [Log partition=kale.marketplace.return.return_noti.de-1, dir=/kafka-data/kafka] Truncating to 1472 has no effect as the largest offset in the log is 1471 (kafka.log.Log)
2021-04-23 08:33:17,183 INFO [Log partition=kale.marketplace.return.return_noti.de-4, dir=/kafka-data/kafka] Truncating to 1448 has no effect as the largest offset in the log is 1447 (kafka.log.Log)
2021-04-23 08:33:21,186 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:21,187 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:21,188 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:21,188 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:21,189 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:21,189 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:22,821 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:22,821 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:22,823 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:22,823 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:22,823 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:22,824 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:24,132 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:24,133 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:24,133 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:24,134 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:24,134 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-6 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:24,134 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:25,417 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:25,417 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:25,418 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:25,418 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:25,418 INFO [Log partition=kale.marketplace.return.return_noti.de-6, dir=/kafka-data/kafka] Truncating to 1461 has no effect as the largest offset in the log is 1460 (kafka.log.Log)
2021-04-23 08:33:25,419 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:26,649 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:26,650 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:26,651 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:26,652 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:26,652 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:28,687 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:28,688 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:28,688 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:28,689 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:28,702 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:31,120 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:31,121 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:31,121 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:31,122 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:31,123 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:33,131 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:33,131 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:33,135 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:33,135 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:33,136 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:34,387 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:34,387 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:34,388 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:34,388 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:34,389 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:36,452 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:36,453 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:36,453 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:36,454 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:36,454 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:37,932 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:37,932 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:37,933 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:37,933 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:37,934 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:39,591 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:39,591 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:39,592 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:39,592 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:39,592 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:41,424 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:41,424 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:41,425 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:41,425 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:41,426 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:43,411 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:43,411 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:43,412 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:43,412 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:43,413 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:46,133 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:46,133 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:46,133 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:46,133 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:46,134 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:48,892 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:48,892 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:48,893 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:48,893 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:48,893 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:50,434 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:50,434 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:50,435 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:50,435 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:50,435 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:51,614 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:51,614 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:51,615 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:51,615 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-14 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:51,615 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:53,914 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:53,914 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:53,914 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:53,914 INFO [Log partition=kale.marketplace.return.return_noti.de-14, dir=/kafka-data/kafka] Truncating to 1541 has no effect as the largest offset in the log is 1540 (kafka.log.Log)
2021-04-23 08:33:53,915 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:56,042 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:56,043 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:56,043 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:56,043 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:57,234 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:57,234 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:57,235 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-11 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:57,235 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:59,891 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:59,891 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:33:59,891 INFO [Log partition=kale.marketplace.return.return_noti.de-11, dir=/kafka-data/kafka] Truncating to 1477 has no effect as the largest offset in the log is 1476 (kafka.log.Log)
2021-04-23 08:33:59,891 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:01,673 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:01,673 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:01,674 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:04,146 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:04,146 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:04,147 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-8 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:06,047 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-5 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:06,047 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:06,048 INFO [Log partition=kale.marketplace.return.return_noti.de-8, dir=/kafka-data/kafka] Truncating to 1480 has no effect as the largest offset in the log is 1479 (kafka.log.Log)
2021-04-23 08:34:08,261 INFO [Log partition=kale.marketplace.return.return_noti.de-5, dir=/kafka-data/kafka] Truncating to 1512 has no effect as the largest offset in the log is 1511 (kafka.log.Log)
2021-04-23 08:34:08,261 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:10,196 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:12,432 INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Retrying leaderEpoch request for partition kale.marketplace.return.return_noti.de-2 as the leader reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
2021-04-23 08:34:13,641 INFO [Log partition=kale.marketplace.return.return_noti.de-2, dir=/kafka-data/kafka] Truncating to 1497 has no effect as the largest offset in the log is 1496 (kafka.log.Log)
2021-04-23 08:34:36,543 INFO [ProducerStateManager partition=kale.marketplace.return.return_noti.de-8] Writing producer snapshot at offset 1480 (kafka.log.ProducerStateManager)
2021-04-23 08:34:36,543 INFO [Log partition=kale.marketplace.return.return_noti.de-8, dir=/kafka-data/kafka] Rolled new log segment at offset 1480 in 1 ms. (kafka.log.Log)
2021-04-23 08:36:00,564 INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(kale.marketplace.order_accounting.state_machine.rt-7, kale.marketplace.order_processing.cancellation.rt-5, kale.marketplace.order_processing.cb_collection.order_sync.de-5, kale.marketplace.order_accounting.tax_service.rt-9, kale.marketplace.return.return_noti.rt-14, kale.marketplace.order_processing.cb_collection.status_sync.de-0, kale.marketplace.order_processing.checkout_expiry.processor.de-1, kale.marketplace.order_processing.auto_cancel.fulfilment_cancel.de-8, kale.marketplace.order_processing.govtech.noti_center.de-8, kale.marketplace.order_processing.noti.email.rt-15, kale.marketplace.order_processing.cancellation.rt-11, kale.marketplace.order_processing.fulfilment.channel.allocation.rt-15, __consumer_offsets-8, kale.marketplace.return.return_processing.de-14, kale.marketplace.order_processing.order_dispatching.de-6, kale.marketplace.order_processing.noti.email.de-11, kale.marketplace.order_processing.noti.email.de-8, kale.marketplace.order_accounting.invoice.region.br.rt-9, kale.marketplace.order_processing.govtech.payment.de-14, kale.marketplace.order_accounting.invoice.region.br.rt-6, kale.marketplace.order_accounting.batch_processing.de-5, kale.marketplace.order_processing.order_logistics.status_update.rt-4, kale.marketplace.order_processing.govtech.payment.de-11, kale.marketplace.return.return_accept.rt-10, kale.marketplace.order_processing.fulfilment.channel.allocation.rt-3, kale.marketplace.order_accounting.batch_processing.de-14, kale.coin.admin.rt-5, kale.order.service_fee.rt-13, kale.coin.admin.rt-2, kale.marketplace.return.return_processing.de-5, kale.marketplace.return.return_negotiation.rt-4, kale.marketplace.order_accounting.receipt.rt-15, kale.marketplace.order_processing.fulfilment.channel.allocation.rt-6, kale.marketplace.order_processing.govtech.order_approval.de-2, kale.marketplace.order_accounting.tax_service.rt-6, kale.marketplace.order_processing.auto_cancel.fulfilment_cancel.de-11, kale.marketplace.order_accounting.invoice_file.rt-6, kale.marketplace.order_accounting.tax_service.rt-15, kale.marketplace.order_processing.govtech.noti_center.de-2, kale.marketplace.order_processing.fulfilment.stock_adjustment.rt-10, kale.marketplace.order_processing.cb_collection.status_sync.de-6, kale.marketplace.order_processing.cancellation.rt-14, kale.coin.admin.rt-8, kale.marketplace.return.return_accept.rt-13, __consumer_offsets-35, kale.marketplace.order_processing.order_dispatching.rt-12, kale.marketplace.order_processing.order_dispatching.de-15, __consumer_offsets-41, __consumer_offsets-23, kale.marketplace.order_accounting.seller_income.rt-11, kale.marketplace.order_processing.auto_cancel.fulfilment_cancel.de-5, kale.marketplace.return.return_noti.de-0, kale.marketplace.order_processing.govtech.order_approval.de-14, kale.marketplace.order_processing.fulfillment_timing.shipbydate.rt-15, kale.marketplace.return.return_compensation.rt-7, kale.marketplace.order_accounting.invoice.region.br.rt-12, kale.marketplace.order_processing.order_dispatching.de-12, kale.marketplace.order_processing.govtech.payment.de-5, kale.marketplace.return.return_processing.rt-4, kale.marketplace.order_processing.govtech.payment.de-2, __consumer_offsets-47, kale.marketplace.order_accounting.invoice_file.rt-15, kale.coin.alert.rt-2, kale.marketplace.return.return_noti.de-9, kale.marketplace.return.return_compensation.rt-10, kale.marketplace.return.return_processing.de-8, kale.marketplace.order_processing.fulfilment.ofg_status_scanner.rt-10, kale.marketplace.order_processing.govtech.noti_center.de-14, kale.marketplace.order_processing.fulfilment.stock_adjustment.rt-4, kale.order.service_fee.rt-10, kale.marketplace.order_processing.order_logistics.post_upd_hdlr.rt-1, kale.promotion.core.de-2, kale.marketplace.order_processing.cb_collection.status_sync.de-3, kale.marketplace.order_processing.fulfilment.ofg_status_scanner.rt-7, kale.coin.admin.rt-14, kale.marketplace.order_processing.checkout_expiry.processor.de-7, kale.marketplace.order_processing.order_dispatching.de-0, kale.marketplace.order_processing.order_logistics.post_upd_hdlr.rt-7, kale.marketplace.order_processing.cb_collection.order_sync.de-14, kale.marketplace.order_accounting.batch_processing.de-8, kale.marketplace.order_accounting.state_machine.rt-1, kale.marketplace.order_accounting.receipt.rt-12, kale.marketplace.order_processing.noti.email.rt-3, kale.marketplace.return.return_negotiation.rt-1, kale.marketplace.order_processing.cod_screening.de-12, kale.marketplace.order_processing.noti.email.de-14, kale.marketplace.order_processing.fulfillment_timing.shipbydate.rt-12, kale.marketplace.order_processing.govtech.order_approval.de-5, kale.marketplace.order_processing.fulfilment.ofg_status_scanner.rt-4, kale.marketplace.return.return_accept.rt-4, kale.marketplace.order_processing.govtech.noti_center.de-11, kale.coin.alert.rt-14, kale.marketplace.order_processing.fulfilment.channel.allocation.rt-9, kale.marketplace.order_processing.fulfilment.ofg_status_scanner.rt-13, kale.marketplace.return.return_noti.de-3, kale.marketplace.order_processing.govtech.payment.de-8, kale.marketplace.return.return_processing.de-11, kale.marketplace.order_processing.cancellation.rt-2, kale.marketplace.order_processing.order_logistics.post_upd_hdlr.rt-13, kale.promotion.core.de-5, kale.marketplace.order_processing.fulfillment_timing.shipbydate.rt-9, kale.marketplace.order_processing.order_dispatching.rt-0, kale.order.service_fee.rt-7, kale.marketplace.return.return_accept.rt-7, kale.marketplace.order_processing.fulfilment.ofg_status_scanner.rt-1, kale.coin.admin.rt-11, kale.marketplace.order_processing.cb_collection.status_sync.de-9, kale.marketplace.order_accounting.state_machine.rt-10, kale.marketplace.order_processing.cb_collection.logistics_sync.de-2, kale.marketplace.order_processing.fulfillment_timing.shipbydate.rt-0, __consumer_offsets-38, kale.marketplace.order_processing.auto_cancel.fulfilment_cancel.de-14, kale.marketplace.order_accounting.tax_service.de-2, __consumer_offsets-17, kale.marketplace.order_processing.cb_collection.order_sync.de-2, kale.marketplace.return.return_processing.rt-7, kale.marketplace.order_accounting.invoice.region.br.de-7, kale.marketplace.order_accounting.tax_service.de-11, kale.marketplace.order_accounting.invoice.region.br.rt-0, kale.marketplace.order_processing.noti.email.de-2, kale.marketplace.order_processing.govtech.noti_center.de-5, kale.marketplace.order_processing.cod_screening.de-9, kale.marketplace.return.return_negotiation.rt-7, __consumer_offsets-11, kale.marketplace.order_processing.order_logistics.post_upd_hdlr.rt-4, kale.marketplace.order_accounting.seller_income.rt-8, kale.marketplace.order_processing.cancellation.rt-8, kale.marketplace.order_processing.cb_collection.order_sync.de-8, kale.marketplace.order_processing.fulfillment_timing.shipbydate.rt-3, kale.marketplace.return.return_noti.rt-8, kale.marketplace.order_accounting.receipt.rt-9, kale.marketplace.order_processing.order_logistics.status_update.rt-10, kale.marketplace.order_accounting.seller_income.rt-5, kale.marketplace.order_accounting.invoice_file.rt-12, kale.coin.alert.rt-8, kale.promotion.core.de-11, kale.marketplace.order_processing.order_logistics.post_upd_hdlr.rt-10, kale.marketplace.return.return_negotiation.rt-13, kale.marketplace.order_processing.order_dispatching.rt-9, kale.marketplace.order_processing.cod_screening.de-15, kale.coin.alert.rt-11, kale.marketplace.order_accounting.invoice.region.br.de-13, kale.marketplace.order_processing.fulfilment.stock_adjustment.rt-7, kale.marketplace.order_processing.noti.email.rt-6, __consumer_offsets-2, kale.marketplace.order_processing.cb_collection.status_sync.de-15, kale.marketplace.order_accounting.invoice_file.rt-3, kale.marketplace.order_processing.cb_collection.logistics_sync.de-5, kale.marketplace.order_accounting.seller_income.rt-2, __consumer_offsets-14, kale.marketplace.order_processing.govtech.order_approval.de-11, kale.marketplace.order_processing.cod_screening.de-3, kale.order.service_fee.rt-4, kale.marketplace.order_accounting.invoice.region.br.rt-3, kale.marketplace.return.return_noti.de-15, kale.marketplace.order_processing.order_logistics.status_update.rt-7, kale.marketplace.order_processing.order_logistics.status_update.rt-13, kale.marketplace.order_processing.cb_collection.logistics_sync.de-11, kale.marketplace.order_accounting.seller_income.rt-14, kale.marketplace.order_processing.order_dispatching.de-9, kale.marketplace.order_accounting.tax_service.de-8, kale.marketplace.return.return_processing.rt-1, kale.marketplace.return.return_noti.de-6, kale.marketplace.order_accounting.batch_processing.de-11, kale.marketplace.return.return_processing.rt-13, kale.marketplace.return.return_processing.rt-10, kale.marketplace.return.return_compensation.rt-13, kale.marketplace.order_accounting.batch_processing.de-2, kale.marketplace.return.return_compensation.rt-1, kale.marketplace.order_processing.auto_cancel.fulfilment_cancel.de-2, kale.marketplace.order_processing.fulfilment.stock_adjustment.rt-13, kale.marketplace.order_processing.fulfilment.channel.allocation.rt-12, kale.marketplace.order_processing.noti.email.de-5, kale.marketplace.order_accounting.invoice.region.br.rt-15, kale.marketplace.order_accounting.tax_service.rt-3, kale.marketplace.order_accounting.invoice.region.br.de-10, kale.marketplace.order_processing.cb_collection.order_sync.de-11, __consumer_offsets-20, kale.marketplace.order_accounting.receipt.rt-3, __consumer_offsets-44, kale.marketplace.order_processing.checkout_expiry.processor.de-10, kale.marketplace.order_processing.noti.email.rt-0, kale.promotion.core.de-8, kale.marketplace.return.return_noti.rt-2, kale.marketplace.order_processing.checkout_expiry.processor.de-13, kale.marketplace.order_processing.order_dispatching.rt-15, kale.marketplace.order_processing.noti.email.rt-12, kale.marketplace.order_accounting.tax_service.rt-12, kale.marketplace.order_processing.order_dispatching.de-3, kale.marketplace.order_processing.cod_screening.de-0, kale.marketplace.return.return_negotiation.rt-10, kale.marketplace.order_accounting.invoice.region.br.de-1, kale.marketplace.order_processing.noti.email.rt-9, kale.marketplace.order_processing.fulfilment.stock_adjustment.rt-1, kale.order.service_fee.rt-1, kale.marketplace.order_processing.cb_collection.status_sync.de-12, __consumer_offsets-5, __consumer_offsets-26, kale.marketplace.order_accounting.invoice.region.br.de-4, kale.marketplace.order_accounting.state_machine.rt-13, kale.marketplace.return.return_noti.rt-5, kale.marketplace.order_processing.cb_collection.logistics_sync.de-8, kale.marketplace.order_processing.fulfillment_timing.shipbydate.rt-6, kale.marketplace.order_processing.cb_collection.logistics_sync.de-14, __consumer_offsets-29, kale.promotion.core.de-14, kale.marketplace.order_accounting.tax_service.de-14, kale.marketplace.order_accounting.receipt.rt-0, kale.marketplace.order_processing.govtech.order_approval.de-8, kale.marketplace.order_processing.checkout_expiry.processor.de-4, kale.marketplace.order_accounting.state_machine.rt-4, kale.marketplace.order_accounting.invoice_file.rt-9, kale.coin.alert.rt-5, __consumer_offsets-32, kale.marketplace.order_accounting.tax_service.rt-0, kale.marketplace.order_processing.order_dispatching.rt-6, kale.marketplace.order_processing.order_logistics.status_update.rt-1, kale.marketplace.order_processing.cod_screening.de-6, kale.marketplace.order_accounting.invoice_file.rt-0, kale.marketplace.order_processing.order_dispatching.rt-3, kale.marketplace.return.return_noti.de-12, kale.marketplace.return.return_noti.rt-11, kale.marketplace.order_accounting.receipt.rt-6, kale.marketplace.order_processing.fulfilment.channel.allocation.rt-0, kale.marketplace.order_accounting.tax_service.de-5, kale.marketplace.return.return_compensation.rt-4, kale.marketplace.return.return_processing.de-2, kale.marketplace.return.return_accept.rt-1) (kafka.server.ReplicaFetcherManager)
2021-04-23 08:36:01,138 INFO [Partition kale.marketplace.return.return_noti.de-12 broker=0] kale.marketplace.return.return_noti.de-12 starts at Leader Epoch 29 from offset 1429. Previous Leader Epoch was: 28 (kafka.cluster.Partition)
2021-04-23 08:36:01,396 INFO [Partition kale.marketplace.return.return_noti.de-3 broker=0] kale.marketplace.return.return_noti.de-3 starts at Leader Epoch 29 from offset 1448. Previous Leader Epoch was: 28 (kafka.cluster.Partition)
2021-04-23 08:36:02,711 INFO [Partition kale.marketplace.return.return_noti.de-0 broker=0] kale.marketplace.return.return_noti.de-0 starts at Leader Epoch 29 from offset 1418. Previous Leader Epoch was: 28 (kafka.cluster.Partition)
2021-04-23 08:36:22,909 INFO [Partition kale.marketplace.return.return_noti.de-9 broker=0] kale.marketplace.return.return_noti.de-9 starts at Leader Epoch 27 from offset 1473. Previous Leader Epoch was: 26 (kafka.cluster.Partition)
2021-04-23 08:36:26,917 INFO [Partition kale.marketplace.return.return_noti.de-6 broker=0] kale.marketplace.return.return_noti.de-6 starts at Leader Epoch 27 from offset 1461. Previous Leader Epoch was: 26 (kafka.cluster.Partition)
2021-04-23 08:36:28,900 INFO [Partition kale.marketplace.return.return_noti.de-15 broker=0] kale.marketplace.return.return_noti.de-15 starts at Leader Epoch 27 from offset 1496. Previous Leader Epoch was: 26 (kafka.cluster.Partition)

hoobean1996 commented 3 years ago

it seems the client group stuck in such a loop.

rd_kafka_cgrp_coord_query  -> 
rd_kafka_FindCoordinatorRequest ->
rd_kafka_cgrp_handle_FindCoordinator ->
rd_kafka_cgrp_serve -> 
some state to trig rd_kafka_cgrp_coord_query again

may I how could I locate which branch to go rd_kafka_cgrp_coord_query again? thank you :)

edenhill commented 3 years ago

Your cluster seems to be in an unavailable state where the client's FindCoordinator requests returns broker 0 has group coordinator, but broker 0 is returning "Not coordinator" for any group coordinator requests. There isn't much the client can do about this.

confluentinc / confluent-kafka-go

Consumer Stuck for some reason #638

Description

How to reproduce

Checklist