This set of changes implements more nuanced handling of Delta vs. message schema with the introduction of the -S command line flag which enables schema evolutions.
In order for schema evolution to work, there is a necessary performance hit. kafka-delta-ingest must determine the schema of every message that is read from Kafka, infer its schema, and if necessary add nullable columns to the Delta table. This is related to similar work in delta-rs for the RecordBatchWriter but deviates because of the mechanism by which RecordBatches and schema is handled in kafka-delta-ingest.
This set of changes implements more nuanced handling of Delta vs. message schema with the introduction of the
-Scommand line flag which enables schema evolutions.In order for schema evolution to work, there is a necessary performance hit. kafka-delta-ingest must determine the schema of every message that is read from Kafka, infer its schema, and if necessary add nullable columns to the Delta table. This is related to similar work in delta-rs for the
RecordBatchWriterbut deviates because of the mechanism by which RecordBatches and schema is handled in kafka-delta-ingest.Sponsored-by: Raft LLC
NOTE: This pull request builds on #162