search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
823 stars 163 forks source link

Expose CopyExec metrics in Spark #935

Open andygrove opened 2 months ago

andygrove commented 2 months ago

What is the problem the feature request solves?

In the native planner, we insert a CopyExec node into the plan. The timings from this operator are not visible in the Spark UI.

Describe the potential solution

I think there are two choices:

  1. Create a JVM equivalent CopyExec and make this available in the plan in Spark
  2. Have CopyExec include its time in its child operators metrics

Additional context

No response