confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
41 stars 1.04k forks source link

Automatically terminate internal C*AS query on DROP STREAM|TABLE #5987

Closed spena closed 3 years ago

spena commented 3 years ago

Is your feature request related to a problem? Please describe.

Partial request for KLIP-20: Remove TERMINATE:

tl;dr: Requiring users to TERMINATE a persistent query before dropping a source causes friction in 
the user experience. The id of the query(s) that need terminating are not determinism or accessible 
through SQL. We propose dropping the source should automatically drop persistent query(s) that 
write to it.

This feature request is part of the ksqlDB migrations to allow users to drop streams and tables from SQL scripts. Users cannot access the query ID from SQL scripts, so it is impossible to do ksqlDB migrations without it.

Describe the solution you'd like

DROP STREAM <stream> | TABLE <table> should terminate the internal persistent query running in the background without requiring the user to call the TERMINATE command.

Constraints If another query (stream/table or insert/select) is reading or writing on the same stream/table, then the DROP command should fail and not allow the user to terminate and drop the stream. This is a similar behavior from other DBs systems when having a foreign key constraint between tables. If one table has a reference, then dropping the table is not possible until the other table is dropped too.

This should avoid the issues mentioned in the KLIP-20:

We cannot drop a stream/table if a query is writing into it. Otherwise we will have queries that write 
into zombie stream/tables!

Describe alternatives you've considered

The current alternative is to use TERMINATE to terminate the query, then call DROP to drop the stream or table. But ksqlDB migrations are not possible with this.

Additional context

Similar requests:

spena commented 3 years ago

Marking this as duplicate. Will track the work in https://github.com/confluentinc/ksql/issues/2177