search

dyrnq / Stream-Data

This is a project for some basic big data exercise.
0 stars 0 forks source link

./bin/connect-distributed.sh config/connect-distributed.properties #21

Open dyrnq opened 1 year ago

dyrnq commented 1 year ago 
./bin/connect-distributed.sh config/connect-distributed.properties
dyrnq commented 1 year ago

kafka.log

dyrnq commented 1 year ago 
[2023-04-27 08:47:18,707] INFO [kafka-console-sink|task-8] [Consumer clientId=connector-consumer-kafka-console-sink-8, groupId=connect-kafka-console-sink] Adding newly assigned partitions:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,707] INFO [kafka-console-sink|task-9] [Consumer clientId=connector-consumer-kafka-console-sink-9, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=34, memberId='connector-consumer-kafka-console-sink-9-4193d384-1d19-4867-862f-4a2335a950a6', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 08:47:18,707] INFO [kafka-console-sink|task-9] [Consumer clientId=connector-consumer-kafka-console-sink-9, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 08:47:18,707] INFO [kafka-console-sink|task-9] [Consumer clientId=connector-consumer-kafka-console-sink-9, groupId=connect-kafka-console-sink] Adding newly assigned partitions:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,708] INFO [kafka-console-sink|task-4] [Consumer clientId=connector-consumer-kafka-console-sink-4, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=34, memberId='connector-consumer-kafka-console-sink-4-e877f0d2-e117-44a1-bedb-29611ac87dce', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 08:47:18,708] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=34, memberId='connector-consumer-kafka-console-sink-0-cb041bfa-ea38-4e57-85a3-90d88ba66666', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 08:47:18,708] INFO [kafka-console-sink|task-4] [Consumer clientId=connector-consumer-kafka-console-sink-4, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 08:47:18,708] INFO [kafka-console-sink|task-4] [Consumer clientId=connector-consumer-kafka-console-sink-4, groupId=connect-kafka-console-sink] Adding newly assigned partitions:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,708] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[mytopic-0]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 08:47:18,712] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Adding newly assigned partitions: mytopic-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,718] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=34, memberId='connector-consumer-kafka-console-sink-1-e554b5d4-5aa5-432e-a0c5-0638f8e032f5', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 08:47:18,719] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[mytopic-1]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 08:47:18,719] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Adding newly assigned partitions: mytopic-1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,715] INFO [kafka-console-sink|task-7] [Consumer clientId=connector-consumer-kafka-console-sink-7, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=34, memberId='connector-consumer-kafka-console-sink-7-1c7db9de-8216-49e9-915f-81eb606db0e0', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 08:47:18,713] INFO [kafka-console-sink|task-3] [Consumer clientId=connector-consumer-kafka-console-sink-3, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=34, memberId='connector-consumer-kafka-console-sink-3-e7b8edaa-54b4-46dd-8b72-1f314a2d0171', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 08:47:18,721] INFO [kafka-console-sink|task-7] [Consumer clientId=connector-consumer-kafka-console-sink-7, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 08:47:18,721] INFO [kafka-console-sink|task-7] [Consumer clientId=connector-consumer-kafka-console-sink-7, groupId=connect-kafka-console-sink] Adding newly assigned partitions:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,721] INFO [kafka-console-sink|task-3] [Consumer clientId=connector-consumer-kafka-console-sink-3, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 08:47:18,721] INFO [kafka-console-sink|task-3] [Consumer clientId=connector-consumer-kafka-console-sink-3, groupId=connect-kafka-console-sink] Adding newly assigned partitions:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 08:47:18,723] INFO [kafka-console-sink|task-2] [Consumer clientId=connector-consumer-kafka-console-sink-2, groupId=connect-kafka-console-sink] Setting offset for partition mytopic-2 to the committed offset FetchPosition{offset=121, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.6.213:9092 (id: 13 rack: null)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:973)
[2023-04-27 08:47:18,723] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Setting offset for partition mytopic-1 to the committed offset FetchPosition{offset=119, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.6.213:9092 (id: 13 rack: null)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:973)
[2023-04-27 08:47:18,723] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Setting offset for partition mytopic-0 to the committed offset FetchPosition{offset=75, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.6.213:9092 (id: 13 rack: null)], epoch=2}} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:973)
[2023-04-27 08:52:17,463] INFO [AdminClient clientId=connect-cluster--shared-admin] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:17,643] INFO [Consumer clientId=connect-cluster--offsets, groupId=connect-cluster] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:17,718] INFO [Producer clientId=connect-cluster--offsets] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,027] INFO [Consumer clientId=connect-cluster--statuses, groupId=connect-cluster] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,047] INFO [Producer clientId=connect-cluster--statuses] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,152] INFO [Consumer clientId=connect-cluster--configs, groupId=connect-cluster] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,179] INFO [Worker clientId=connect-1, groupId=connect-cluster] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,263] INFO [Producer clientId=connect-cluster--configs] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,512] INFO [kafka-console-sink|task-7] [Consumer clientId=connector-consumer-kafka-console-sink-7, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,535] INFO [kafka-console-sink|task-5] [Consumer clientId=connector-consumer-kafka-console-sink-5, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,551] INFO [kafka-console-sink|task-4] [Consumer clientId=connector-consumer-kafka-console-sink-4, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,608] INFO [kafka-console-sink|task-3] [Consumer clientId=connector-consumer-kafka-console-sink-3, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,624] INFO [kafka-console-sink|task-6] [Consumer clientId=connector-consumer-kafka-console-sink-6, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,631] INFO [kafka-console-sink|task-8] [Consumer clientId=connector-consumer-kafka-console-sink-8, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,725] INFO [kafka-console-sink|task-9] [Consumer clientId=connector-consumer-kafka-console-sink-9, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,736] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,736] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:18,737] INFO [kafka-console-sink|task-2] [Consumer clientId=connector-consumer-kafka-console-sink-2, groupId=connect-kafka-console-sink] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:56:48,080] INFO [Producer clientId=connect-cluster--statuses] Node 11 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:57:17,566] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 12 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 08:57:17,567] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 11 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:01:18,156] INFO [Consumer clientId=connect-cluster--configs, groupId=connect-cluster] Node 13 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:01:18,315] INFO [Producer clientId=connect-cluster--statuses] Node 12 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:02:17,660] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 13 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:07:17,761] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 12 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:12:17,861] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 11 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:17:17,957] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 13 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:22:18,061] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 12 disconnected. (org.apache.kafka.clients.NetworkClient:937)
dyrnq commented 1 year ago 
[2023-04-27 08:45:42,532] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-4-57b417ae-1be3-4b41-89aa-2ed52842e204, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-4, clientHost=/192.168.6.212, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 29. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,532] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-9-1a94b890-70dd-4956-9516-e90acb666f1a, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-9, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 29. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,532] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-2-c5b0be6e-6b24-40ac-a3a2-ca5b4ecd41f0, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-2, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 29. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,532] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-3-402b2eaf-809d-49b8-a3f2-ac054f4a34ae, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-3, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 29. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,532] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-7-433a5d35-0c53-4982-8e94-66bdb40a14b6, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-7, clientHost=/192.168.6.212, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 29. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,533] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-5-7b0c8afb-14de-478c-90fe-ef3c110b400d, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-5, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 31. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,533] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-8-ada769c0-109a-4054-873a-868066aac183, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-8, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 31. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,533] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-9-1a94b890-70dd-4956-9516-e90acb666f1a, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-9, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 31. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,534] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-2-c5b0be6e-6b24-40ac-a3a2-ca5b4ecd41f0, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-2, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 31. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,534] INFO Loaded member MemberMetadata(memberId=connector-consumer-kafka-console-sink-3-402b2eaf-809d-49b8-a3f2-ac054f4a34ae, groupInstanceId=None, clientId=connector-consumer-kafka-console-sink-3, clientHost=/192.168.6.213, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group connect-kafka-console-sink with generation 31. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,540] INFO [GroupCoordinator 11]: Loading group metadata for connect-kafka-console-sink with generation 32 (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:45:42,543] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-21 in 278 milliseconds for epoch 2, of which 208 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,544] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-24 in 279 milliseconds for epoch 2, of which 278 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,544] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-27 in 278 milliseconds for epoch 2, of which 278 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,544] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-30 in 278 milliseconds for epoch 2, of which 278 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,545] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-33 in 279 milliseconds for epoch 2, of which 279 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,549] INFO Loaded member MemberMetadata(memberId=console-consumer-241cee79-3ada-4522-86c6-6b13f381387b, groupInstanceId=None, clientId=console-consumer, clientHost=/192.168.6.211, sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, supportedProtocols=List(range)) in group console-consumer-84959 with generation 1. (kafka.coordinator.group.GroupMetadata$)
[2023-04-27 08:45:42,550] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-36 in 284 milliseconds for epoch 2, of which 279 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,550] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-39 in 284 milliseconds for epoch 2, of which 284 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,551] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-42 in 285 milliseconds for epoch 2, of which 285 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,558] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-45 in 285 milliseconds for epoch 2, of which 285 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:45:42,560] INFO [GroupMetadataManager brokerId=11] Finished loading offsets and group metadata from __consumer_offsets-48 in 294 milliseconds for epoch 2, of which 293 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-27 08:47:18,506] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in Empty state. Created a new member id connector-consumer-kafka-console-sink-7-1c7db9de-8216-49e9-915f-81eb606db0e0 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,532] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in Empty state. Created a new member id connector-consumer-kafka-console-sink-5-383fd396-a01d-4fc2-8585-b923e42bb8b2 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,562] INFO [GroupCoordinator 11]: Preparing to rebalance group connect-kafka-console-sink in state PreparingRebalance with old generation 32 (__consumer_offsets-21) (reason: Adding new member connector-consumer-kafka-console-sink-7-1c7db9de-8216-49e9-915f-81eb606db0e0 with group instance id None; client reason: rebalance failed due to MemberIdRequiredException) (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,563] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-4-e877f0d2-e117-44a1-bedb-29611ac87dce and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,590] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-6-d24f6f62-417a-4429-a29f-32a914c628b5 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,591] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-3-e7b8edaa-54b4-46dd-8b72-1f314a2d0171 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,599] INFO [GroupCoordinator 11]: Stabilized group connect-kafka-console-sink generation 33 (__consumer_offsets-21) with 3 members (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,616] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in CompletingRebalance state. Created a new member id connector-consumer-kafka-console-sink-8-c7b364f0-5ef7-41ae-be23-0a6ab3ec9d61 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,626] INFO [GroupCoordinator 11]: Preparing to rebalance group connect-kafka-console-sink in state PreparingRebalance with old generation 33 (__consumer_offsets-21) (reason: Adding new member connector-consumer-kafka-console-sink-3-e7b8edaa-54b4-46dd-8b72-1f314a2d0171 with group instance id None; client reason: rebalance failed due to MemberIdRequiredException) (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,628] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-1-e554b5d4-5aa5-432e-a0c5-0638f8e032f5 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,633] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-0-cb041bfa-ea38-4e57-85a3-90d88ba66666 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,642] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-2-359915ce-bff8-471c-ae7f-925708c4039b and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,642] INFO [GroupCoordinator 11]: Dynamic member with unknown member id joins group connect-kafka-console-sink in PreparingRebalance state. Created a new member id connector-consumer-kafka-console-sink-9-4193d384-1d19-4867-862f-4a2335a950a6 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,667] INFO [GroupCoordinator 11]: Stabilized group connect-kafka-console-sink generation 34 (__consumer_offsets-21) with 10 members (kafka.coordinator.group.GroupCoordinator)
[2023-04-27 08:47:18,685] INFO [GroupCoordinator 11]: Assignment received from leader connector-consumer-kafka-console-sink-7-1c7db9de-8216-49e9-915f-81eb606db0e0 for group connect-kafka-console-sink for generation 34. The group has 10 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
dyrnq commented 1 year ago 
[2023-04-27 09:47:18,538] INFO [Producer clientId=connect-cluster--configs] Resetting the last seen epoch of partition connect-configs-0 to 2 since the associated topicId changed from null to AMDIUCaJTcOHPj89Vg8OFw (org.apache.kafka.clients.Metadata:402)
[2023-04-27 09:47:18,546] INFO [AdminClient clientId=connect-cluster--shared-admin] Node 13 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-04-27 09:47:18,559] INFO [Worker clientId=connect-1, groupId=connect-cluster] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:2151)
dyrnq commented 1 year ago 
[2023-04-27 11:37:23,085] WARN [kafka-console-sink|task-1] These configurations '[metrics.context.connect.kafka.cluster.id, metrics.context.connect.group.id]' were supplied but are not used yet. (org.apache.kafka.clients.consumer.ConsumerConfig:385)
[2023-04-27 11:37:23,085] INFO [kafka-console-sink|task-1] Kafka version: 3.4.0 (org.apache.kafka.common.utils.AppInfoParser:119)
[2023-04-27 11:37:23,086] INFO [kafka-console-sink|task-1] Kafka commitId: 2e1947d240607d53 (org.apache.kafka.common.utils.AppInfoParser:120)
[2023-04-27 11:37:23,086] INFO [kafka-console-sink|task-1] Kafka startTimeMs: 1682566643085 (org.apache.kafka.common.utils.AppInfoParser:121)
[2023-04-27 11:37:23,091] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1705)
[2023-04-27 11:37:23,093] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Subscribed to topic(s): mytopic (org.apache.kafka.clients.consumer.KafkaConsumer:969)
[2023-04-27 11:37:23,095] INFO [kafka-console-sink|task-1] AbstractConfig values: 
    file = null
 (org.apache.kafka.common.config.AbstractConfig:376)
[2023-04-27 11:37:23,095] INFO [kafka-console-sink|task-1] WorkerSinkTask{id=kafka-console-sink-1} Sink task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:316)
[2023-04-27 11:37:23,095] INFO [kafka-console-sink|task-1] WorkerSinkTask{id=kafka-console-sink-1} Executing sink task (org.apache.kafka.connect.runtime.WorkerSinkTask:201)
[2023-04-27 11:37:23,100] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Resetting the last seen epoch of partition mytopic-0 to 2 since the associated topicId changed from null to XyLst5w0S32OAdqVYj7J1Q (org.apache.kafka.clients.Metadata:402)
[2023-04-27 11:37:23,101] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Resetting the last seen epoch of partition mytopic-2 to 3 since the associated topicId changed from null to XyLst5w0S32OAdqVYj7J1Q (org.apache.kafka.clients.Metadata:402)
[2023-04-27 11:37:23,101] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Resetting the last seen epoch of partition mytopic-1 to 3 since the associated topicId changed from null to XyLst5w0S32OAdqVYj7J1Q (org.apache.kafka.clients.Metadata:402)
[2023-04-27 11:37:23,101] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Cluster ID: kVfz6b8LQEeFUtSZSt4pLQ (org.apache.kafka.clients.Metadata:287)
[2023-04-27 11:37:23,102] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Discovered group coordinator 192.168.6.211:9092 (id: 2147483636 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:896)
[2023-04-27 11:37:23,102] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:566)
[2023-04-27 11:37:23,105] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Request joining group due to: need to re-join with the given member-id: connector-consumer-kafka-console-sink-1-694bc96e-daed-445f-adbf-9438a8b92db8 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1062)
[2023-04-27 11:37:23,105] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1062)
[2023-04-27 11:37:23,106] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:566)
[2023-04-27 11:37:26,103] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Successfully joined group with generation Generation{generationId=39, memberId='connector-consumer-kafka-console-sink-1-694bc96e-daed-445f-adbf-9438a8b92db8', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:627)
[2023-04-27 11:37:26,109] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=39, memberId='connector-consumer-kafka-console-sink-1-694bc96e-daed-445f-adbf-9438a8b92db8', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 11:37:26,109] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[mytopic-2]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 11:37:26,110] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Adding newly assigned partitions: mytopic-2 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 11:37:26,111] INFO [kafka-console-sink|task-1] [Consumer clientId=connector-consumer-kafka-console-sink-1, groupId=connect-kafka-console-sink] Setting offset for partition mytopic-2 to the committed offset FetchPosition{offset=127, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.6.213:9092 (id: 13 rack: null)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:973)
dyrnq commented 1 year ago 
[2023-04-27 11:40:03,718] WARN [kafka-console-sink|task-0] These configurations '[metrics.context.connect.kafka.cluster.id, metrics.context.connect.group.id]' were supplied but are not used yet. (org.apache.kafka.clients.consumer.ConsumerConfig:385)
[2023-04-27 11:40:03,718] INFO [kafka-console-sink|task-0] Kafka version: 3.4.0 (org.apache.kafka.common.utils.AppInfoParser:119)
[2023-04-27 11:40:03,718] INFO [kafka-console-sink|task-0] Kafka commitId: 2e1947d240607d53 (org.apache.kafka.common.utils.AppInfoParser:120)
[2023-04-27 11:40:03,718] INFO [kafka-console-sink|task-0] Kafka startTimeMs: 1682566803718 (org.apache.kafka.common.utils.AppInfoParser:121)
[2023-04-27 11:40:03,719] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1705)
[2023-04-27 11:40:03,720] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Subscribed to topic(s): mytopic (org.apache.kafka.clients.consumer.KafkaConsumer:969)
[2023-04-27 11:40:03,720] INFO [kafka-console-sink|task-0] AbstractConfig values: 
    file = null
 (org.apache.kafka.common.config.AbstractConfig:376)
[2023-04-27 11:40:03,720] INFO [kafka-console-sink|task-0] WorkerSinkTask{id=kafka-console-sink-0} Sink task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:316)
[2023-04-27 11:40:03,721] INFO [kafka-console-sink|task-0] WorkerSinkTask{id=kafka-console-sink-0} Executing sink task (org.apache.kafka.connect.runtime.WorkerSinkTask:201)
[2023-04-27 11:40:03,723] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Resetting the last seen epoch of partition mytopic-0 to 2 since the associated topicId changed from null to XyLst5w0S32OAdqVYj7J1Q (org.apache.kafka.clients.Metadata:402)
[2023-04-27 11:40:03,724] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Resetting the last seen epoch of partition mytopic-2 to 3 since the associated topicId changed from null to XyLst5w0S32OAdqVYj7J1Q (org.apache.kafka.clients.Metadata:402)
[2023-04-27 11:40:03,724] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Resetting the last seen epoch of partition mytopic-1 to 3 since the associated topicId changed from null to XyLst5w0S32OAdqVYj7J1Q (org.apache.kafka.clients.Metadata:402)
[2023-04-27 11:40:03,724] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Cluster ID: kVfz6b8LQEeFUtSZSt4pLQ (org.apache.kafka.clients.Metadata:287)
[2023-04-27 11:40:03,724] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Discovered group coordinator 192.168.6.211:9092 (id: 2147483636 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:896)
[2023-04-27 11:40:03,725] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:566)
[2023-04-27 11:40:03,728] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Request joining group due to: need to re-join with the given member-id: connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1062)
[2023-04-27 11:40:03,728] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1062)
[2023-04-27 11:40:03,729] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:566)
[2023-04-27 11:40:03,732] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Successfully joined group with generation Generation{generationId=41, memberId='connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:627)
[2023-04-27 11:40:03,732] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Finished assignment for group at generation 41: {connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49=Assignment(partitions=[mytopic-0, mytopic-1, mytopic-2])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:710)
[2023-04-27 11:40:03,734] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=41, memberId='connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:826)
[2023-04-27 11:40:03,735] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1062)
[2023-04-27 11:40:03,735] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:566)
[2023-04-27 11:40:03,737] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Successfully joined group with generation Generation{generationId=42, memberId='connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:627)
[2023-04-27 11:40:03,737] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Finished assignment for group at generation 42: {connector-consumer-kafka-console-sink-1-8609275a-cd39-4297-b926-db476643793a=Assignment(partitions=[mytopic-2]), connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49=Assignment(partitions=[mytopic-0, mytopic-1])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:710)
[2023-04-27 11:40:03,740] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Successfully synced group in generation Generation{generationId=42, memberId='connector-consumer-kafka-console-sink-0-22aa5149-69ea-48ec-8195-0e87acc87d49', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:802)
[2023-04-27 11:40:03,740] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Notifying assignor about the new Assignment(partitions=[mytopic-0, mytopic-1]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:305)
[2023-04-27 11:40:03,740] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Adding newly assigned partitions: mytopic-0, mytopic-1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:317)
[2023-04-27 11:40:03,742] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Setting offset for partition mytopic-0 to the committed offset FetchPosition{offset=80, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.6.213:9092 (id: 13 rack: null)], epoch=2}} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:973)
[2023-04-27 11:40:03,742] INFO [kafka-console-sink|task-0] [Consumer clientId=connector-consumer-kafka-console-sink-0, groupId=connect-kafka-console-sink] Setting offset for partition mytopic-1 to the committed offset FetchPosition{offset=126, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[192.168.6.213:9092 (id: 13 rack: null)], epoch=3}} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:973)
tooptoop4 commented 9 months ago

did u solve @dyrnq ?

dyrnq commented 9 months ago

did u solve @dyrnq ?

@tooptoop4 upgrade jvm from jdk8 to jdk11