search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
77 stars 1.04k forks source link

What's the purpose of the "Project-Last-changelog" internal topic? #9316

Closed alonpr closed 2 years ago

alonpr commented 2 years ago

What's the purpose behind the *-Project-Last-changelog internal topic in table-table joins? It looks like writing into it happens just before writing into the sink topic so what state store basically this changelog topic backs?

suhas-satish commented 2 years ago

@mjsax , do you know ?

mjsax commented 2 years ago

The sink topic cannot be used as changelog (it's a limitation of Kafka Streams: https://issues.apache.org/jira/browse/KAFKA-6035).

Thus, the topic in question is the changelog of the final result of the query. (Closing this. Feel free to follow up if you have more questions.)

alonpr commented 2 years ago

@mjsax I'm just trying to wrap my head around it, why the final result of the query needs a changelog? Is the sink topic considered by Streams as a state store? And if so, is it for the purpose of during a recovery to recover the final result of the query also from a changelog instead of re-running the query on recovered state stores upstream in the topology?