[Spark] Column mapping removal: support streaming tables

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.17k stars 1.62k forks source link

[Spark] Column mapping removal: support streaming tables #3061

Closed sabir-akhadov closed 1 month ago

sabir-akhadov commented 1 month ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Add support for streaming reads on tables that undergo column mapping downgrade. Basically most streams should fail when a downgrade is happening on the table.

How was this patch tested?

New unit tests We test all the different scenarios also with schema tracking enabled.

Does this PR introduce any user-facing changes?

sabir-akhadov commented 1 month ago

@tomvanbussel could you review this one please?

tomvanbussel commented 1 month ago

Could you also add [Spark] to the title of the PR?