confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
87 stars 1.04k forks source link

Force terminate query #4381

Open mac2000 opened 4 years ago

mac2000 commented 4 years ago

I'm running confluentinc/cp-ksql-server:5.3.2 connected to confluent.cloud

After playing with different queries I somehow broke ksql server

I do have an ksqldemo3, when trying to delete it with drop table ksqldemo3; I receive expected error complaing that there is a query writing to it.

But when I try to terminate query CTAS_KSQLDEMO3_3; ksql cli hangs

In ksql server logs I did found only one related message:

INFO stream-client [_..._query_CTAS_KSQLDEMO3_3-...] Already in the pending shutdown state, wait to complete shutdown (org.apache.kafka.streams.KafkaStreams:850)

I did tried to restart ksql server but query still here

Yes I can change service id of ksql server to start over but it not going to work later on when it will be in production

Reading the docs it seems that there is no force option for terminating a query which might be helpfull in such situation

jeffreymlutz commented 4 years ago

FYI. I figured out how to delete a query that typically hangs and never returns from the command "terminate ".
1) Checkout the logs of all of the running ksql servers. Most likely, the backing Kafka doesn't exist and therefore an error in the log file will repeat. If this is the case, then add the Kafka topic back into Kafka 2) Confirm the error in the ksql server logs regarding the missing Kafka topic is no longer echoed in the logs. 3) Restart ksql servers. If running ksql servers in Kubernetes: "kubectl delete pod/ksql..." 4) The ksql servers show restart and then the "terminate " command will work.

Apparently, if the Kafka topic is deleted while a query is running, ksql servers can no longer control the query. So, don't delete a topic for a running ksql query before deleting the query first.