estuary / connectors

Connectors for capturing data from external data sources
Other
47 stars 13 forks source link

CDC Connectors history mode & deletions #1070

Closed psFried closed 4 months ago

psFried commented 9 months ago

The idea is to have an endpoint configuration option that enables "history mode". When this option is enabled, the connector will add an associative: false annotation to discovered JSON schemas.

jgraettinger commented 8 months ago

Adding to scope: add delete annotations on _meta/op d events to thread through end-to-end deletions.

jgraettinger commented 5 months ago

Currently, associative: false is only available for the lastWriteWins reduction annotation.

I've been trying to recall whether that was fully intentional, or whether it really ought to be available for the merge annotation as well. It doesn't seem like a bad idea for it to work with merge. But I am noting just now, that it would also work fine to pick a sub-location within the larger document (such as /_meta/op and mark that as {reduce: {strategy: lastWriteWins, associative: false}}.

This works because any _sub_location which is non-associative will also cause the entire document to not be combined associatively at capture (only fully reduced at materialization).