search

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
337 stars 72 forks source link

How to ingest the key? #117

Closed rymail closed 2 years ago

rymail commented 2 years ago

I have looked in the repo but could not find the configuration to include the key while ingesting. Can someone help with this please?

Sample message in my topic: student_id,{"first_name":"john","last_name":"doe","courses":["math","english"]} I want to write all these fields (including the key student_id) to the Delta table.

rymail commented 2 years ago

I figured it out, I had to modify transforms.rs file to include the key. Maybe a candidate for enhancement.