search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
826 stars 164 forks source link

chore: Enable Comet shuffle for Spark core-1 test #924

Closed viirya closed 2 months ago

viirya commented 2 months ago

Which issue does this PR close?

Closes #930.

Rationale for this change

We disabled Comet shuffle for Spark core-1 test because negative ref count issue reported by Java Arrow. After fixing memory issues and some changes to more follow C Data interface, we may re-enable Comet shuffle for Spark core-1 test in CI.

What changes are included in this PR?

How are these changes tested?

codecov-commenter commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 34.15%. Comparing base (033fe6f) to head (9fb03f1). Report is 13 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #924 +/- ## ============================================ + Coverage 34.03% 34.15% +0.11% + Complexity 883 882 -1 ============================================ Files 113 112 -1 Lines 43170 43237 +67 Branches 9516 9562 +46 ============================================ + Hits 14693 14766 +73 + Misses 25471 25455 -16 - Partials 3006 3016 +10 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

viirya commented 2 months ago

Seems negative ref count issue was fixed by previous commits revising array import/export.

viirya commented 2 months ago

lgtm thanks @viirya 1 minor why we have RUST_BACKTRACE enabled for ansi workflow test and disabled for regular one?

Thanks @comphead

I'm not sure about it. Maybe we have added it for debugging ansi workflow before?