An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
We currently support schemaTrackingLocation (doc) that allows Delta streaming source to track additive and non-additive schema changes during streaming from a Delta table.
Right now, every single schema changes would cause a new schema version to be generated in the tracking location, however, it might not be necessary for non-backward incompatible changes like ADD COLUMN. One way is to speculate ahead in the Delta log, skip over the ADD COLUMN schema changes, and use the encompassing schema as the stream read schema.
Motivation
This could allow possibly a lot fewer stream restarts due to ADD COLUMN schema changes.
Further details
The speculation ahead can be potentially implemented here
Willingness to contribute
The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?
[ ] Yes. I can contribute this feature independently.
[ ] Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
[ ] No. I cannot contribute this feature at this time.
Feature request
Which Delta project/connector is this regarding?
Overview
We currently support
schemaTrackingLocation(doc) that allows Delta streaming source to track additive and non-additive schema changes during streaming from a Delta table.Right now, every single schema changes would cause a new schema version to be generated in the tracking location, however, it might not be necessary for non-backward incompatible changes like ADD COLUMN. One way is to speculate ahead in the Delta log, skip over the ADD COLUMN schema changes, and use the encompassing schema as the stream read schema.
Motivation
This could allow possibly a lot fewer stream restarts due to ADD COLUMN schema changes.
Further details
The speculation ahead can be potentially implemented here
Willingness to contribute
The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?