NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
764 stars 225 forks source link

[FEA] `ExecutionPlanCaptureCallback` should support capturing multiple plans #6229

Closed andygrove closed 4 months ago

andygrove commented 1 year ago

Is your feature request related to a problem? Please describe. ExecutionPlanCaptureCallback is used in tests to capture query plans and allow them to be inspected but it currently only supports capturing a single query.

When executing queries against Delta Lake tables, the Delta Lake implementation executes many metadata queries behind the scenes, and we currently have no way of capturing those in tests to confirm whether they were executed as intended (such as falling back to CPU).

Describe the solution you'd like I propose that we enhance ExecutionPlanCaptureCallback to support capturing multiple queries.

Describe alternatives you've considered None

Additional context None

jlowe commented 4 months ago

Implemented by #7342