search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
5.88k stars 1.04k forks source link

Deleted topics will show again. #2884

Open xs005 opened 5 years ago

xs005 commented 5 years ago

I am using the Kafka stack of 5.1.2, but my KSQL is 5.2.1 because it has many new functions. I got two major problems that I cannot find solutions online.

  1. When I tried to terminate a query, there CLI did not have any repose and the worst thing was all KSQL-connected Kafka brokers would restart automatically. I thought the computing-intensive work should be on the KSQL server instead of Kafka brokers, but KSQL server was fine and only Kafka brokers restarted.

  2. I have many topics generated by KStream and KTable. After testing, I would terminate related queries, then drop streams or tables, and then delete the topics in Kafka Manager. I can check the folders in Kafka brokers of these deleted topics were gone. However, if I restarted KSQL server, those deleted topics would show again with 0 offset.

Could someone give me some suggestion? KSQL is a cool tool. I'd like to use it for more apps.

agavra commented 5 years ago

Hello @xs005 - the behavior in the first point is unexpected. Terminating a query should have nothing to do with the kafka brokers. Can you look through the KSQL broker log and see if anything interesting shows up there?

For the second point, this is a known bug and is fixed in 5.3 (stay tuned for the release in the next few weeks)! A temporary fix for this would be to use DROP FOO DELETE TOPIC instead of manually deleting the topic. Then when you restart the server it will re-create the topic, and then re-delete the topic (still not ideal, but short term fix until 5.3).

xs005 commented 5 years ago

@agavra Thanks for your reply. I find the PR #1217 about the delete topics. I will add the error log of KSQL server later.

holgerbrandl commented 4 years ago

@agavra Is ksql supposed to drop topics created by ksql when dropping streams? This would be the most intuitive behaviour because otherwise, an interactive development (where streams are setup multiple times until the correct solution has been created) becomes very tricky. However, it does not seem to do so in v5.3.1: topics are kept after DROP STREAM foo;

EDIT Please ignore my question, the description of #1217 answers my questions. Still, I think the default (not deleting the stream without any indication that the topic is kept and will be reused if the stream is recreated) is confusing for new users.

agavra commented 4 years ago

@holgerbrandl - your concern is totally valid, I agree that it can be confusing. The reason I think it makes sense to depart from this "intuitive" behavior, however, is that the risk of accidentally deleting a topic is high. If you DROP STREAM foo without realizing that the topic will be deleted, you could be causing unrecoverable harm.

Perhaps it makes sense to also issue a note "topic will not be deleted" or something like that.