search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.33k stars 411 forks source link

fix: column parsing to include nested columns and enclosing char #2737

Closed gtrawinski closed 3 months ago

gtrawinski commented 3 months ago

Description

Added column parsing to include nested columns and `` enclosing to adopt 0.18.x

Addressing issue

2572

https://github.com/delta-io/delta/blob/4b102d34a2ce881b2a851b4c6cfbf2ab3ab5534f/spark/src/main/scala/org/apache/spark/sql/delta/DeltaConfig.scala#L549-L561

Changes:

Added:

_extract_enclosedstring - is a utility function to remove enclosing characters from a string. _get_statsfield - is a function to retrieve a field from a schema, handling nested field names and removing enclosing backticks.

alexwilcoxson-rel commented 3 months ago

Is meant to address https://github.com/delta-io/delta-rs/issues/2572

https://github.com/delta-io/delta/blob/4b102d34a2ce881b2a851b4c6cfbf2ab3ab5534f/spark/src/main/scala/org/apache/spark/sql/delta/DeltaConfig.scala#L549-L561