search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
39 stars 1.04k forks source link

KSQLDB - ERROR "Written offsets would not be recorded and no more records would be sent since this is a fatal error" #8080

Open victor-ugarte-mojix-com opened 2 years ago

victor-ugarte-mojix-com commented 2 years ago

Hello everyone, I have got the following errors in KSQLDB since I have upgraded the Kafka version from 2.6 to 2.8. The errors appeared with the version 0.13.0 of KSQLDB.

First, It was thought it was a problem related to the upgrade, then we have removed Ksqldb and all the topics related to it, then we reployed it but the errors still appeared in logs.

Later, it was desided to upgrade the Ksqldb version to 0.19.0. Then, after the deployment and after the creation of streams/tables, the error continue to appear in the logs.

Bellow you will find the errors obtained, any comment or help will be appreciated.

[2021-08-26 19:41:38,192] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3] task [0_2] Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,214] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3] task [0_2] Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,197] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_7] Failed to flush cache of store KsqlTopic-Reduce:  (org.apache.kafka.streams.processor.internals.ProcessorStateManager:497)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,196] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3] task [0_2] Failed to flush cache of store KsqlTopic-Reduce:  (org.apache.kafka.streams.processor.internals.ProcessorStateManager:497)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,195] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_7] Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,234] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_7] Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,246] ERROR Unhandled exception caught in streams thread _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3. (UNKNOWN) (io.confluent.ksql.util.QueryMetadataImpl:197)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:41:38,246] ERROR Unhandled exception caught in streams thread _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4. (UNKNOWN) (io.confluent.ksql.util.QueryMetadataImpl:197)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:08,252] INFO Restarting query CTAS_TENANT_KSQL_PRODUCT_75 (attempt #2) (io.confluent.ksql.util.QueryMetadataImpl:437)
[2021-08-26 19:42:08,258] ERROR {"type":4,"deserializationError":null,"recordProcessingError":null,"productionError":null,"serializationError":null,"kafkaStreamsThreadError":{"errorMessage":"Unhandled exception caught in streams thread","threadName":"_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3","cause":["Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:\norg.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.\nWritten offsets would not be recorded and no more records would be sent since this is a fatal error.","The server experienced an unexpected error when processing the request."]}} (processing.CTAS_TENANT_KSQL_PRODUCT_75.ksql.logger.thread.exception.uncaught:44)
[2021-08-26 19:42:08,281] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3] Error flushing caches of dirty task 0_2  (org.apache.kafka.streams.processor.internals.TaskManager:754)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Exception in thread "_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-3" org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_2 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:38,265] INFO Restarting query CTAS_TENANT_KSQL_PRODUCT_75 (attempt #2) (io.confluent.ksql.util.QueryMetadataImpl:437)
[2021-08-26 19:42:38,267] ERROR {"type":4,"deserializationError":null,"recordProcessingError":null,"productionError":null,"serializationError":null,"kafkaStreamsThreadError":{"errorMessage":"Unhandled exception caught in streams thread","threadName":"_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4","cause":["Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:\norg.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.\nWritten offsets would not be recorded and no more records would be sent since this is a fatal error.","The server experienced an unexpected error when processing the request."]}} (processing.CTAS_TENANT_KSQL_PRODUCT_75.ksql.logger.thread.exception.uncaught:44)
[2021-08-26 19:42:38,303] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] Error flushing caches of dirty task 0_7  (org.apache.kafka.streams.processor.internals.TaskManager:754)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:38,348] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_3] Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:38,349] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_3] Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:38,348] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_3] Failed to flush cache of store KsqlTopic-Reduce:  (org.apache.kafka.streams.processor.internals.ProcessorStateManager:497)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:38,351] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] Error flushing caches of dirty task 0_3  (org.apache.kafka.streams.processor.internals.TaskManager:754)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:38,372] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4] task [0_3] Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Exception in thread "_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-4" org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_7 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:48,034] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-5] task [0_3] Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error. (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:234)
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:48,036] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-5] task [0_3] Failed to flush cache of store KsqlTopic-Reduce:  (org.apache.kafka.streams.processor.internals.ProcessorStateManager:497)
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-KsqlTopic-Reduce-changelog for task 0_3 due to:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
Written offsets would not be recorded and no more records would be sent since this is a fatal error.
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:209)
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:196)
    at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1369)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751)
    at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:703)
    at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:642)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:583)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:570)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:570)
    at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:844)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
    at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
[2021-08-26 19:42:48,047] ERROR Unhandled exception caught in streams thread _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_75-1e48b6dd-4ba5-4284-8431-ed915774ce9c-StreamThread-5. (UNKNOWN) (io.confluent.ksql.util.QueryMetadataImpl:197)
vcrfxia commented 2 years ago

Hey @victor-ugarte-mojix-com , the error message from the logs you've shared means the Streams producer previously encountered an error and will no longer be sending future records because of it. The log snippet shared does not contain what that previous error was, though. Do you see another error before the errors in the log snippet above?

victor-ugarte-mojix-com commented 2 years ago

Thank you for your answer @vcrfxia , well, before the first error we only got logs with INFO level. I post it below:

(It is important to mention, It only happens when streaming data has started)

[2021-09-07 19:11:04,177] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,178] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,181] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,186] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,186] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,199] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,177] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,223] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,216] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,226] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,222] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] State transition from STARTING to PARTITIONS_ASSIGNED (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,231] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,640] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,642] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,648] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,654] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] task [0_5] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-5 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,657] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] task [0_5] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,656] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,657] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] task [0_7] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-7 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,657] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] task [0_7] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,654] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] task [0_1] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-1 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,661] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] task [0_1] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,654] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] task [0_3] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-3 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,661] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] task [0_3] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,707] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,714] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,707] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,716] INFO Opening store KsqlTopic-Reduce in regular mode (org.apache.kafka.streams.state.internals.RocksDBTimestampedStore:100) [2021-09-07 19:11:04,738] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] task [0_6] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-6 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,739] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] task [0_6] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,741] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] task [0_2] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-2 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,742] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] task [0_2] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,738] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] task [0_4] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-4 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,742] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] task [0_4] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,741] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] task [0_0] State store KsqlTopic-Reduce did not find checkpoint offset, hence would default to the starting offset at changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-0 (org.apache.kafka.streams.processor.internals.ProcessorStateManager:256) [2021-09-07 19:11:04,744] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] task [0_0] Initialized (org.apache.kafka.streams.processor.internals.StreamTask:246) [2021-09-07 19:11:04,789] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-3 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,793] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-5 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,797] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-1 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,802] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-7 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,809] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] task [0_5] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,809] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] Restoration took 646 ms for all tasks [0_5] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,809] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,810] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] task [0_7] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,811] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] Restoration took 633 ms for all tasks [0_7] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,812] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-4] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,813] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] task [0_1] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,814] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] Restoration took 636 ms for all tasks [0_1] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,815] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,815] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] task [0_3] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,815] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] Restoration took 633 ms for all tasks [0_3] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,815] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,816] INFO stream-client [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0] State transition from REBALANCING to RUNNING (org.apache.kafka.streams.KafkaStreams:318) [2021-09-07 19:11:04,873] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-2 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,874] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,879] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-6 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,883] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-4 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,883] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] Finished restoring changelog _confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-KsqlTopic-Reduce-changelog-0 to store KsqlTopic-Reduce with a total number of 0 records (org.apache.kafka.streams.processor.internals.StoreChangelogReader:609) [2021-09-07 19:11:04,883] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] Setting topic 'TENANTviewproduct' to consume from earliest offset (org.apache.kafka.streams.processor.internals.StreamThread:1001) [2021-09-07 19:11:04,904] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] task [0_6] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,904] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] Restoration took 679 ms for all tasks [0_6] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,904] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-4] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,905] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] task [0_4] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,907] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] task [0_2] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,910] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] Restoration took 686 ms for all tasks [0_2] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,910] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,911] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] Restoration took 684 ms for all tasks [0_4] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,911] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,915] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] task [0_0] Restored and ready to run (org.apache.kafka.streams.processor.internals.StreamTask:271) [2021-09-07 19:11:04,915] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] Restoration took 686 ms for all tasks [0_0] (org.apache.kafka.streams.processor.internals.StreamThread:856) [2021-09-07 19:11:04,915] INFO stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING (org.apache.kafka.streams.processor.internals.StreamThread:230) [2021-09-07 19:11:04,916] INFO stream-client [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-4a99594b-a406-4adb-b987-1f048a4a879d] State transition from REBALANCING to RUNNING (org.apache.kafka.streams.KafkaStreams:318) [2021-09-07 19:11:05,109] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-03b949b6-f06b-471d-a74a-b87bb6fa4061-StreamThread-3] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,166] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-03b949b6-f06b-471d-a74a-b87bb6fa4061-StreamThread-1] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,204] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-03b949b6-f06b-471d-a74a-b87bb6fa4061-StreamThread-2] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,211] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-03b949b6-f06b-471d-a74a-b87bb6fa4061-StreamThread-4] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,217] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-986a8638-53ed-4a3f-bf77-ed79924cbd67-StreamThread-4] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,223] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-986a8638-53ed-4a3f-bf77-ed79924cbd67-StreamThread-2] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,224] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-986a8638-53ed-4a3f-bf77-ed79924cbd67-StreamThread-1] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:05,236] INFO stream-thread [_confluent-ksql-default_query_CSAS_TENANT_RECEIVING_KSQL_EVENTS_REFLIST_PRODUCT_QTY_TEMP_S1_117-986a8638-53ed-4a3f-bf77-ed79924cbd67-StreamThread-3] Processed 0 total records, ran 0 punctuators, and committed 0 total tasks since the last update (org.apache.kafka.streams.processor.internals.StreamThread:832) [2021-09-07 19:11:07,076] ERROR stream-thread [_confluent-ksql-default_query_CTAS_TENANT_KSQL_PRODUCT_77-500ff56a-605c-4e68-9e46-32daec257bd0-StreamThread-3] task [0_5] Error encountered sending record to topic TENANT_KSQL_PRODUCT for task 0_5 due to: