delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

CDC is not generated for Structs and Lists #2568

Open rtyler opened 3 weeks ago

rtyler commented 3 weeks ago

This is a placeholder issue for future reference in code warnings.

In short, there is an issue upstream in Datafusion and its Arrow dependency which prevents comparisons happening properly when the Delta table schema contains StructType or ListType, and therefore the CDC code in delta-rs cannot properly compute the before/after difference on a set of data.

Once apache/datafusion#10749 is resolved and released, then we can remove the warning.