search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.85k stars 3.77k forks source link

changefeedccl: consider standarizing the output format via cloudevents #42179

Open lopezator opened 4 years ago

lopezator commented 4 years ago

Offering native support for cloudevents in addition of JSON/Avro would be a great addition in order to standarize and improve interoperability with other services and platforms.

https://github.com/cloudevents/spec

CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms and systems.

The project already reached the 1.0 version as stated here:

https://www.cncf.io/announcement/2019/10/28/serverless-specification-cloudevents-reaches-version-1-0/

This event spec is being pushed natively in other popular solutions:

CloudEvents v1.0 has already been implemented in Knative’s Eventing framework, Red Hat’s EventFlow, Eclipse Vert.x and Debezium, SAP’s Kyma, and Serveless.com’s Event Gateway, among many others. Soon after the spec was created, Microsoft announced their support for CloudEvents natively for all events in Azure, via Azure Event Grid.

Others are being proposed as "adapters":

https://github.com/cloudevents/spec/tree/master/adapters

A lot of big actors are supporting this unified model:

https://github.com/cloudevents/spec/blob/master/community/contributors.md

You can find the proof of concept I made here:

https://github.com/lopezator/cockroachdb-cdc/tree/master/cc/ce-ps

Looking into the example of the couchdb example

A draft example spec for CockroachDB CDC could look similar to:

CloudEvents Attribute Value
id The event identifier
source The server URL / db
specversion 1.0
type com.cockroachlabs.createchangefeed.update
datacontenttype application/json
subject The table name
time Current time or Updated time if using (WITH UPDATED)
data Json contents of the CDC row

Thoughts?

CC/ @rolandcrosby

Epic CRDB-2260

Jira issue: CRDB-5381

github-actions[bot] commented 11 months ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!

lopezator commented 11 months ago

This still would be a great addition for us. Any plans to add this to the backlog sometime or is it discarded at all @amruss ?

Thanks!