databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

Place CDC metadata in struct/map in SMTs #116

Closed bryanck closed 1 year ago

bryanck commented 1 year ago

This PR updates the CDC schema to put metadata in a struct/map field named _cdc, in the DMS and Debezium SMTs. For Debezium, the stucture of the record will look something like:

{
    "id": 123
    "col1": "abc",
    "col2": "xyz",
    "_cdc": {
      "op": "U",
      "ts": 12312312313,
      "source": "public.accounts",
      "target": "db.accounts_cdc",
          "offset": 123,
      "key": {
        "id": 123
      }
    }
}
okayhooni commented 1 year ago

I am really happy to see those developing features related to Debezium CDC format record supports..!

Thanks a lot...!