apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

Support `GetArrayStructFields` #992

Closed Kimahriman closed 1 month ago

Kimahriman commented 1 month ago

What is the problem the feature request solves?

The last support for struct and array extraction is the GetArrayStructFields expression, which is an optimization to select a nested struct field inside an array of structs, which turns into an array of the inner field type.

Describe the potential solution

Create a custom physical expression to support this. Datafusion doesn't currently support this, but it can be implemented very efficiently with the way arrow lists and structs are setup.

Additional context

No response