search

sanjuthomas / kafka-connect-gcp-bigtable

Kafka Sink Connect to GCP Bigtable - https://www.confluent.io/hub/sanjuthomas/kafka-connect-gcp-bigtable
http://sanjuthomas.com
MIT License
7 stars 8 forks source link

Support for transformation of KSQL Structs. #11

Open nbyrnes-acv opened 5 years ago

nbyrnes-acv commented 5 years ago

I am trying to sink our KSQL streams into BigTable. We have a relatively complex JSON schema (nested arrays and objects). We can use the big query connector from Wepay to sink into big query.

Having never written any Kafka Connectors before, is it possible to replace the Transformer provided in your code, with an object from the Wepay connector, or, is the Transformer specific to this connector's sink target?

sanjuthomas commented 5 years ago

Yes, you can write your own transform and change the config accordingly.

https://github.com/sanjuthomas/kafka-connect-gcp-bigtable/blob/master/etc/demo-topic.yml#L7

Your transformer should comply to Transformer<SinkRecord, WritableRow>

Please let me know if you need help with it.