search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
127 stars 1.04k forks source link

ksql tables not starting on version 0.29.0 #10046

Closed florian-psaier closed 1 year ago

florian-psaier commented 1 year ago

Describe the bug i updated ksql from 0.28.2 to 0.29.0 on k8s. ksql-streams continue working but the ksql-tables did not resume, they seem to be trapped in a startup loop. the tables are windowed without joins. moving back to 0.28.2 did resolve the issue, but now i cannot add new queries as the server status is "DEGRADED" giving the following error: "WARNING: The server has encountered an incompatible entry in its log and cannot process further DDL statements. This is most likely due to the service being rolled back to an earlier version." Any one having a similar issue or a solution? Thanks

To Reproduce one of the tables that will not start is named "DATA_15MINUTES". some logs are following: log file

Expected behavior The tables should be able to continue working.

Actual behaviour Seems to be stuck in resuming the tables.

Additional context Add any other context about the problem here.

florian-psaier commented 1 year ago

after several restarts it started working now (creating new pods)

suhas-satish commented 1 year ago

Its possible that there were some bad records that were incompatible with the new version that just expired due to TTL and hence your problem got solved with that.

florian-psaier commented 1 year ago

the table stopped working again on 0.29.0, when reverting to 0.28.2 it starts to work again.

florian-psaier commented 1 year ago

now I am having the same issue with v0.28.2 it seems the issue is resolved randomly when restarting the service multiple times, but than hangs again after a specific amount of time.

I think the issue could be related to rebalancing, but honestly I have no idea.

[2023-08-24 07:54:54,821] INFO stream-thread [_confluent-ksql-ksql-units-2query_CTAS_UNIT_METRICS_15MINUTES_17-5b871bb0-a8e6-4d69-8428-efcc462921d9-StreamThread-4] at state PENDING_SHUTDOWN: partitions [unit_metrics-0, unit_metrics-6, unit_metrics-9] lost due to missed rebalance.

the kafka cluster has 3 brokers, I am adding more logs, help is very much appreciated, thank you.

the ksql-table I collected logs for is named "UNIT_METRICS_15MINUTES" and consumes messages from the topic "unit_metrics".

unit_metrics_15minutes_error_v0.28.2_2.log

florian-psaier commented 1 year ago

the issue was related to a UDF throwing an exception. once I isolated the table on a own ksqldb instance I was able to capture the error in the logs.