search

an0r0c / kafka-connect-transform-tojsonstring

transform-to-json-string is a Single Message Transformation (SMT) for Apache Kafka® Connect to convert a given Connect Record to a single JSON String. It's an UNOFFICIAL community project.
Apache License 2.0
32 stars 12 forks source link

how do you specify which field should be transformed #4

Closed sgandon closed 1 year ago

sgandon commented 2 years ago

Hello,

I am interested by your connector in order to transform an avro array of records with 2 fields. 1/ Do you think you transformer could help me in that regard. 2/ How do you specify which field to transform to json because I just want this array in json and not the whole record.

an0r0c commented 2 years ago

Hi!

If you want to express this avro array as a json sting in the target system this SMT might be a solution for your problem.

At the moment the SMT takes the whole value and transforms it a json string.

If you need to just one specific sub-part of your records schema you can easily combine this SMT with other SMTs. Like with replacField to drop all other fields before you transform it to a json (https://rmoff.net/2020/12/21/twelve-days-of-smt-day-10-replacefield/)

bpetersen commented 1 year ago

Hello @an0r0c , how might I accomplish this without losing any fields? Is there a way to use existing SMTs and this transform to achieve that? Basically, I have a schema with some regular fields and a field that's complex that I'd like to transform. Then using the JDBC sink, put that into a PostgreSQL table with a few primitive columns and a JSON column for that one complex field.

Any thoughts would be greatly appreciated.

an0r0c commented 1 year ago

Hi,

I think that is not possible with the current implementation. It takes the whole value structure and "maps" it to a simple schema that contains one simple string field and you can specify the name of this field.