cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.5k stars 3.7k forks source link

changefeedccl: breaking behavior change between pubsub v1 and v2 #123888

Open rharding6373 opened 1 month ago

rharding6373 commented 1 month ago

In pubsub v1, changefeeds with json encoders emitted events of the form {key: ..., value: ..., topic: ...} (all lowercase). In pubsub v2, this format was changed to {Key: ..., Value: ..., Topic: ...} (capitalized), which is a breaking change. Specifically, if customers are upgrading from 23.1, where pubsub v2 is disabled by default, or an earlier version to 23.2 or later, where pubsub v2 is enabled by default, downstream systems can break because they're unable to parse the new syntax.

We should either:

  1. change pubsub v2 to use all lower case (which may cause a breaking change for customers already on 23.2)
  2. create an option for pubsub to emit all lower case or capitalized keys
  3. update documentation with a warning that this breaking change will happen when either upgrading to 23.2 or setting the cluster setting changefeed.new_pubsub_sink_enabled to true.

Jira issue: CRDB-38589

blathers-crl[bot] commented 1 month ago

cc @cockroachdb/cdc

rharding6373 commented 1 month ago

We should also consider that we'd like to deprecate older sink versions, and whatever option we choose here could affect our ability to deprecate the old version of pubsub (if customers prefer to remain on the old syntax).

rharding6373 commented 1 month ago

Downgrading priority to P-3, since we now have documentation to warn users about this breaking change. Keeping this open to consider making the naming consistent across sinks.