Closed andygrove closed 36 minutes ago
@mbutrovich I'd be interested in your review on my hack to include ScanExec metrics in ShuffleWriterExec
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 55.08%. Comparing base (
b74bfe4
) to head (5f86bc9
). Report is 8 commits behind head on main.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
🚨 Try these New Features:
Which issue does this PR close?
N/A
Rationale for this change
Provide more visibility into shuffle costs. We already report shuffle write time but we could not see how long it took to read the input data. This PR adds a new
jvm_fetch_time
metric.What changes are included in this PR?
Add a new custom metric to show the time that it takes for native shuffle to read its input data (fetching batches from the JVM that already exist).
How are these changes tested?
Manually.