search

an0r0c / kafka-connect-transform-tojsonstring

transform-to-json-string is a Single Message Transformation (SMT) for Apache Kafka® Connect to convert a given Connect Record to a single JSON String. It's an UNOFFICIAL community project.
Apache License 2.0
32 stars 13 forks source link

Option for keeping columns out of being transformed #13

Closed gschmutz closed 1 year ago

gschmutz commented 1 year ago

Hi

Have an other problem where I would like to hear your opinion. I have the following structure, where I have created the extra "ID" field using a com.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Value before the com.github.cedelsb.kafka.connect.smt.Record2JsonStringConverter$Value SMT:

{
"order": 
    {"id": "04d420f7-5151-4cbb-bca7-773712e57e06"
    , "orderDate": "2022-12-05T10:30:34.106Z"
    , "customerId": -8473150557168245322
    , "orderLines": [{"lineId": 1, "productId": -8526281795280451265, "quantity": 10}, {"lineId": 2, "productId": 6137389060232731205, "quantity": 5}, {"lineId": 3, "productId": -932150108975370299, "quantity": 1}]
    }
, "id": "04d420f7-5151-4cbb-bca7-773712e57e06"
}

I wanted to use the extra id field as the primary key in the JDBC sink connector. But of course the Record2JsonStringConverter SMT is taking the whole struct and transforms it into a single field.

Now I'm wondering if you would support adding an option to "blacklist" some fields not be included in the transformation and by that left as is. It could be named ...keep.fields. Or do you see any other way to do it?

Having this id field besides the "jsonstring" would allow me to use the upsert mode together with the pk.fields option of the JdbcSinkConnector.

Again I could try to implement it and send you a PR.

gschmutz commented 1 year ago

Ok, typical Monday error ;-) of course my problem with the id can easily be solved using the pk.mode=record_key. So I'm closing this issue. Such an option to keep fields might be interesting for other cases, but it would also make your SMT more complex ....