big-andy-coates
commented
3 years ago IMHO, this is a P1, but adding needs-triage label so priority can be discussed.
The related https://github.com/confluentinc/ksql/issues/6438 issue covers fixing the same hole in the new HTTP2 api.
An alternative to fixing this issue on the websocket endpoint would be to decommission this endpoint. Having three ways to subscribe to a query, i.e. HTTP POST, Web Sockets and new HTTP2 API, means any enhancements have to be done in three places. This slows us down. Better to choose one, (the new HTTP2 API) and decommission the others.
The CLI uses the HTTP POST api and would need updating to use HTTP2. (My view would be that the CLI exclusively uses the new API, via the Java client, for all things, enhancing the new API where necessary to support the functionality of the CLI - this means we're dog-fooding the java client and api). The UI uses the Web Socket api and would need updating to use HTTP2.
A task to enable tombstones and keys in the query responses sent over the old websocket API. See https://github.com/confluentinc/ksql/issues/3616 and https://github.com/confluentinc/ksql/issues/6438 for more info.
Importantly, filtering out tombstones means that someone using the ws API can not subscribe to a query and maintain an up-to-date copy of the results. Without the tombstones to indicate when an existing row should be deleted, the number of rows in the result set the client maintains will grow unbounded over time.
UI will need updating to accept them too.