search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
127 stars 1.04k forks source link

KSQL data miss issue #2521

Open salmandream11 opened 5 years ago

salmandream11 commented 5 years ago

We are running KSQL in production with 4 nodes.Today we observed data loss of 10 messages in derived stream (created as CSAS).We got this error in ksql log files. Are these errors in logs expected? and what could be reason for data loss? How to makes sure data doesn't get missed?

[2019-02-17 10:09:53,218] ERROR [Consumer clientId=_confluent-xxxxxxxxxxxxxx-a79f9344-c5a6-47f6-b4b2-fa593559ce53-StreamThread-192-consumer, groupId=_confluent-xxxxxxxxxxxxxx] Offset commit failed on partition topic-0 at offset 51818828: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:843)

[2019-02-17 10:09:53,219] ERROR [Consumer clientId=_confluent-xxxxxxxxxxxxxx-a79f9344-c5a6-47f6-b4b2-fa593559ce53-StreamThread-191-consumer, groupId=_confluent-xxxxxxxxxxxxxx] Offset commit failed on partition topic-4 at offset 51822550: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:843)

[2019-02-17 10:09:53,219] ERROR [Consumer clientId=_confluent-xxxxxxxxxxxxxx-a79f9344-c5a6-47f6-b4b2-fa593559ce53-StreamThread-191-consumer, groupId=_confluent-xxxxxxxxxxxxxx] Offset commit failed on partition topic-4 at offset 51822550: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:843)

[2019-02-17 10:09:53,222] ERROR [Consumer clientId=_confluent-xxxxxxxxxxxxxx-a79f9344-c5a6-47f6-b4b2-fa593559ce53-StreamThread-190-consumer, groupId=_confluent-xxxxxxxxxxxxxx] Offset commit failed on partition topic-2 at offset 51813820: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:843)

[2019-02-17 10:09:53,222] ERROR [Consumer clientId=_confluent-xxxxxxxxxxxxxx-a79f9344-c5a6-47f6-b4b2-fa593559ce53-StreamThread-190-consumer, groupId=_confluent-xxxxxxxxxxxxxx] Offset commit failed on partition topic-2 at offset 51813820: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:843)

[2019-02-17 10:09:53,231] ERROR [Consumer clientId=consumer-1, groupId=] Offset commit failed on partition _confluent-ksql-dream11-ksql-alert_command_topic-0 at offset 109: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:843)

big-andy-coates commented 5 years ago

Hi @salmandream11, the above errors occur when the consumer is trying to store its current offsets with the group coordinator, but the group coordinator has changed which node it is running on in the Kafka cluster. The can occur when the Kafka cluster has nodes join / leave / restart.

The Kafka client will handle this by determining which broker is now hosting the group coordinator and retrying.

This should not cause lost messages. While the client can not sync offsets there is an increased chance of messages being reprocessed, and hence outputs duplicated, should the KSQL nodes fail and restart.

As to why you have lost messages, I can't really say with the information you've provided. Something is definitely happening with the Kafka cluster KSQL is talking too. Otherwise you wouldn't be seeing these errors. Maybe this is the source of your lost messages?