apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

feat: Support `GetArrayStructFields` expression #993

Closed Kimahriman closed 1 month ago

Kimahriman commented 1 month ago

Which issue does this PR close?

Closes #992

Rationale for this change

Support the last complex type extractor for arrays and structs.

What changes are included in this PR?

Adds support for the GetArrayStructFields expression in Spark which extracts a nested struct field of an array of structs as an array of the inner field type.

How are these changes tested?

New expression UT.

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.

Project coverage is 54.84%. Comparing base (c3023c5) to head (7fe6963). Report is 19 commits behind head on main.

Files with missing lines Patch % Lines
.../scala/org/apache/comet/serde/QueryPlanSerde.scala 72.72% 2 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #993 +/- ## ============================================= + Coverage 34.03% 54.84% +20.81% + Complexity 875 858 -17 ============================================= Files 112 109 -3 Lines 43289 10841 -32448 Branches 9572 2101 -7471 ============================================= - Hits 14734 5946 -8788 + Misses 25521 3821 -21700 + Partials 3034 1074 -1960 ``` | [Flag](https://app.codecov.io/gh/apache/datafusion-comet/pull/993/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [](https://app.codecov.io/gh/apache/datafusion-comet/pull/993/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.