search

databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

Not consuming messages from source topic - Kafka consumer errors #196

Open nicodeg87 opened 9 months ago

nicodeg87 commented 9 months ago

Hi guys,

I managed to get the connector working syncing messages from kafka to s3 and after a while I've started to get the following errors on kafka connect worker logs:

[2024-02-21 23:49:17,124] INFO Commit timeout reached (io.tabular.iceberg.connect.channel.CommitState) [2024-02-21 23:49:17,128] INFO Sending event of type: COMMIT_COMPLETE (io.tabular.iceberg.connect.channel.Channel) [2024-02-21 23:49:17,135] INFO Commit 764346e7-b76e-4084-a39a-b3b9a859e427 complete, committed to 0 table(s), vtts null (io.tabular.iceberg.connect.channel.Coordinator) [2024-02-21 23:49:17,135] INFO Sending event of type: COMMIT_REQUEST (io.tabular.iceberg.connect.channel.Channel)

[2024-02-21 23:49:41,442] INFO [Consumer clientId=ec448461-81be-4c2c-8c2c-451b499da90c, groupId=cg-control-3e4da867-b478-47ef-bfd0-308c37c0c985] JoinGroup failed: The coordinator is not aware of this member. Need to re-join the group. Sent generation was Generation{generationId=-1, memberId='ec448461-81be-4c2c-8c2c-451b499da90c-30f57020-bbc1-44e5-a7a8-8457a3bc6d8c', protocol='null'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2024-02-21 23:49:41,442] INFO [Consumer clientId=ec448461-81be-4c2c-8c2c-451b499da90c, groupId=cg-control-3e4da867-b478-47ef-bfd0-308c37c0c985] Resetting generation and member id due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2024-02-21 23:49:41,442] INFO [Consumer clientId=ec448461-81be-4c2c-8c2c-451b499da90c, groupId=cg-control-3e4da867-b478-47ef-bfd0-308c37c0c985] Request joining group due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2024-02-21 23:49:41,443] INFO [Consumer clientId=ec448461-81be-4c2c-8c2c-451b499da90c, groupId=cg-control-3e4da867-b478-47ef-bfd0-308c37c0c985] Request joining group due to: rebalance failed due to 'The coordinator is not aware of this member.' (UnknownMemberIdException) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2024-02-21 23:49:41,443] INFO [Consumer clientId=ec448461-81be-4c2c-8c2c-451b499da90c, groupId=cg-control-3e4da867-b478-47ef-bfd0-308c37c0c985] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2024-02-21 23:49:47,210] INFO Commit timeout reached (io.tabular.iceberg.connect.channel.CommitState) [2024-02-21 23:49:47,223] INFO Sending event of type: COMMIT_COMPLETE (io.tabular.iceberg.connect.channel.Channel) [2024-02-21 23:49:47,239] INFO Commit 819082c4-08fc-4390-b5ec-d7e24e52bf16 complete, committed to 0 table(s), vtts null (io.tabular.iceberg.connect.channel.Coordinator) [2024-02-21 23:49:47,239] INFO Sending event of type: COMMIT_REQUEST (io.tabular.iceberg.conne ct.channel.Channel)

I noticed that the consumer lag keeps increasing and I did not manage to fix it even restarting the connector. Any idea of what could be the reason?

Thanks in advance, Nicolas.

dertodestod commented 8 months ago

I see the same errors in my logs all the time. I understand the UUID consumer group is supposed to be transient but are these messages expected?