apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

chore: Add custom metric for native shuffle fetching batches from JVM #1108

Closed andygrove closed 36 minutes ago

andygrove commented 22 hours ago

Which issue does this PR close?

N/A

Rationale for this change

Provide more visibility into shuffle costs. We already report shuffle write time but we could not see how long it took to read the input data. This PR adds a new jvm_fetch_time metric.

Screenshot from 2024-11-21 09-56-51

What changes are included in this PR?

Add a new custom metric to show the time that it takes for native shuffle to read its input data (fetching batches from the JVM that already exist).

How are these changes tested?

Manually.

andygrove commented 22 hours ago

@mbutrovich I'd be interested in your review on my hack to include ScanExec metrics in ShuffleWriterExec

codecov-commenter commented 5 hours ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 55.08%. Comparing base (b74bfe4) to head (5f86bc9). Report is 8 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1108 +/- ## ============================================= + Coverage 34.33% 55.08% +20.74% + Complexity 898 867 -31 ============================================= Files 115 112 -3 Lines 42986 10872 -32114 Branches 9369 2103 -7266 ============================================= - Hits 14761 5989 -8772 + Misses 25361 3814 -21547 + Partials 2864 1069 -1795 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


🚨 Try these New Features: