Closed HawaiianSpork closed 3 weeks ago
ACTION NEEDED
delta-rs follows the Conventional Commits specification for release automation.
The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.
Please add also a bunch of python tests
Please add also a bunch of python tests
@ion-elgreco I'm confused as to what a Python set of tests would do that the tests in Rust don't already do? :confused:
Please add also a bunch of python tests
@ion-elgreco I'm confused as to what a Python set of tests would do that the tests in Rust don't already do? :confused:
We mostly need to check the reader side with pyarrow datasets
@HawaiianSpork thanks for the PR! can you rebase please so we can merge
@ion-elgreco thank you. Sorry, I did not get around to writing python tests. The code has been rebased.
Description
The current implementation of cast only works for structs nested in structs. This PR adds supports for structs contained in other types (lists and maps). This PR also prevents cast from adding nullable column if the field is not nullable, instead it will throw an error.
Note: This is only a partial solution which would let you merge schema with nested missing columns, it does not allow delta-rs to read the merged schema (though Spark can). To read the merged schema will require another change where delta-rs defines its own datafusion parquet schemaAdapter.