CometNativeExec builds a tree of CometMetric nodes containing SQLMetrics for the Spark plan operators. This structure is passed into the Native.createPlan method.
During native execution, calls to update_metrics will recurse through the native ExecutionPlan tree and through the CometMetric tree simultaneously and copy native metrics to Spark metrics.
This approach assumes a 1:1 mapping between the Spark plan and the native plan. In some cases, this assumption is not correct and we therefore drop metrics and the Spark UI does not show metrics for some nodes.
One example is for aggregate queries, where the native planner sometimes inserts an additional projection.
Another example is that we insert CopyExec nodes around the inputs to aggregate, sort, and join operators.
Describe the bug
CometNativeExecbuilds a tree ofCometMetricnodes containing SQLMetrics for the Spark plan operators. This structure is passed into theNative.createPlanmethod.During native execution, calls to
update_metricswill recurse through the nativeExecutionPlantree and through theCometMetrictree simultaneously and copy native metrics to Spark metrics.This approach assumes a 1:1 mapping between the Spark plan and the native plan. In some cases, this assumption is not correct and we therefore drop metrics and the Spark UI does not show metrics for some nodes.
One example is for aggregate queries, where the native planner sometimes inserts an additional projection.
Another example is that we insert
CopyExecnodes around the inputs to aggregate, sort, and join operators.Steps to reproduce
No response
Expected behavior
No response
Additional context
No response