RedHatInsights / expandjsonsmt

Kafka Connect SMT to expand JSON field
Apache License 2.0
17 stars 18 forks source link

Avro schema with JSON string as value #14

Open SDBrett opened 3 years ago

SDBrett commented 3 years ago

Hi Team,

I'm working on a Kafka solution and am wondering if this transformer is suitable for my use case if a JDBC sink connect plugin is used.

The plugin will consume messages with a AVRO schema but one of the field is a JSON formatted string.

Using the example below I want to extract write the value of customerID from the message field payload.details to a database table. Is this a suitable use case for this transformer?

Example schema

{
    "namespace": "test.example",
    "name": "value",
    "type": "record",
    "fields": [
      {"name": "id", "type": "string"},
      {"name": "payload", "type": {
          "type": "record",
          "name": "payload",
          "fields": [
              {"type": "string","name": "details"}
          ]
        }
      }
    ]
  }

Message Example:

{
    "records": {
        "id": "234234234",
        "payload":{
            "details": '{\"name\":\"myname\", \"customerID\":\"42\"}'
        }
    }
}
Josca commented 3 years ago

@SDBrett I think it should be possible as JSON string extraction is the main purpose of this SMT plugin. Check this test: https://github.com/RedHatInsights/expandjsonsmt/blob/3baa84c29aff29742f646b6e81d6511f61e3d547/src/test/java/com/redhat/insights/expandjsonsmt/ExpandJSONTest.java#L64