search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
116 stars 1.04k forks source link

Table over topic with time based retention #6313

Open big-andy-coates opened 4 years ago

big-andy-coates commented 4 years ago

If a user creates a table (really a materialised view) over a topic with a time based retention, e.g. 1 day, then the data stored in rocks db should ideally also be aged out after roughly the same amount of time, e.g. after about a day. In this way, the data stored in rocks db would roughly equate to the data in the topic.

Currently the data in rocks db would not time out and so can hold records that no longer exist in the topic. This can mean the results of a query can be drastically different before and after a restart, as the restart resets the contents of rocksDB to that in the changelog topic. Aging out the rows in rocks db would mean the difference was negligible.

PeterLindner commented 4 years ago

@big-andy-coates is this related to #3519