confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
108 stars 1.04k forks source link

CSAS stream may be updated with an incompatible query #8301

Open xiangxin72 opened 2 years ago

xiangxin72 commented 2 years ago

Describe the bug Normally speaking, a CSAS stream cannot be updated unless it's backwards compatible and supported as described here (https://docs.ksqldb.io/en/latest/operate-and-deploy/installation/upgrading/).

However, it seems I can go to the Persistent Queries page to terminate the underlying query, and then use CREATE OR REPLACE statement to update the stream with a totally different query, as long as the schema is compatible.

To Reproduce Steps to reproduce the behavior, include:

  1. latest ksql on the cloud
  2. create a new stream: CREATE STREAM foo (WITH ...) AS SELECT * FROM bar WHERE ...
  3. update the stream to a different source and it will be rejected: CREATE OR REPLACE STREAM foo (WITH ...) AS SELECT * FROM baz WHERE ...
  4. terminate the underlying query for foo
  5. rerun step 3 will now succeed

Expected behavior Per definition, CSAS stream cannot be upgraded to a different source stream (step 5 should be rejected)

Actual behaviour A clear and concise description of what actually happens, including: Currently, step 3 will fail with message

Query is not upgradeable. Plan step of type StreamSource must have matching topicName. Values differ: bar vs. baz

but after step 4, rerun will successfully change the source stream to baz

Additional context Incompatible schema is not allowed at step 5, as expected. The attempt to replace will fail with message like:

`Cannot upgrade data source: DataSource '`XX_USERS_TEST`' has schema = `ID` INTEGER KEY, `GENDER` STRING, `NAME` STRING, `AGE` INTEGER which is not upgradeable to `ID` INTEGER KEY, `GENDER` STRING, `NAME` STRING. (The following columns are changed, missing or reordered: [`AGE` INTEGER])
SunQK0515 commented 1 year ago

hi, I have the same problem as you. Do you have a solution?