search

jcustenborder / kafka-connect-json-schema

Apache License 2.0
14 stars 14 forks source link

How to run the transform on field of the document? #13

Open ferozed opened 3 years ago

ferozed commented 3 years ago

I have a dynamo table that I am trying to read. For that i am using the kafka-connect-dynamodb connector from trustpilot.

It creates a record as follows:

{
  "version": "1.0",
  "document": "{\"owner\":\"feroze@daud.com\",\"route\":\"a.177.b22\",\"updateDate\":\"2020-09-03 10:15:30.419751\",\"ticket\":\"DSX-1431\",\"targetSchemaVersion\":1,\"version\":\"1\",\"status\":\"1\"}",
  "source": {
    "version": "1.0",
    "table_name": "route-sync-dev",
    "init_sync": true,
    "init_sync_state": "RUNNING",
    "init_sync_start": 1599518938316,
    "init_sync_end": null,
    "init_sync_count": null
  },
  "op": "r",
  "ts_ms": 1599518938316
}

I want to read from the document field of the result and run the FromJson transform on it. how do I do that? The example in the docs shows it operating on the whole Value field of the kafka record.

marlonpatrick commented 3 years ago

I think you can try to create two transformations: the first with FromJson to read the entire object and transform it into a structured object instead a string. After that, you can use some other transformation to extract just the value you want, for example:

https://docs.confluent.io/current/connect/transforms/extractfield.html

https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ExtractNestedField.html