Streaming Delta Source should not drop NullType columns

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

6.96k stars 1.58k forks source link

Streaming Delta Source should not drop NullType columns #3021

Closed sabir-akhadov closed 2 weeks ago

sabir-akhadov commented 2 weeks ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Prevent createDataFrame from dropping NullType columns with streaming sources.

How was this patch tested?

New unit tests for streaming read/write and test for non-streaming createDataFrame code path verifying the current behavior is preserved.

delta-io / delta

Streaming Delta Source should not drop NullType columns #3021

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?