search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

cdc/changefeed: Include source cluster id and changefeed job id in transport metadata #112880

Open bobvawter opened 11 months ago

bobvawter commented 11 months ago

It would be helpful if the changefeed transport layer could include the source cluster uuid and the changefeed job id (e.g. a request HTTP header for ndjson or webhook deliveries). This would enable changefeed consumers to detect cases where an operator has recreated a feed or "crossed the streams" and fail in a somewhat informative manner when CDC invariants are violated (e.g. resolved timestamps not going backwards).

Jira issue: CRDB-32659

Epic CRDB-37343

gz#21394

blathers-crl[bot] commented 11 months ago

cc @cockroachdb/cdc

amruss commented 11 months ago

Thanks @bobvawter, we'll consider this as part of 24.1 planning

rharding6373 commented 5 months ago

This could be achieved via CDC queries if we added support for functions that return the cluster ID and job ID. The trade off would be that this would be emitted per message instead of per batch, but it provides flexibility to the user.

bobvawter commented 5 months ago

CDC queries, though, are limited to a single table.