search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

Refactor Arrow Array and Schema allocation in ColumnReader and MetadataColumnReader #1048

Closed viirya closed 2 weeks ago

viirya commented 3 weeks ago

Describe the bug

We have refactored Arrow Array and Schema allocation for Comet execution before. Based on Arrow C Data interface spec, the two structures should be allocated by the consumer.

Another place we incorrectly allocate them is in ColumnReader and MetadataColumnReader. Previously the incorrect allocation caused a few strange errors. We should refactor the allocation in ColumnReader and MetadataColumnReader too.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response