[BUG] Fix potential problems and AQE updates in Qual tool - Githubissues

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

44 stars 34 forks source link

[BUG] Fix potential problems and AQE updates in Qual tool #1019

Closed amahussein closed 1 month ago

amahussein commented 1 month ago

Describe the bug

As a followup on #1000 , there are some remaining issues to be addressed:

Qual tool generate an empty column "potential problems" in the sql_duration_and_executor_cpu_time_percent.csv file
There is an old bug in the Qualifiction tool where. when AQE overrides an existing SQLPlanInfo, it overrides its value in the SQLPlan map. However, it won't cleanup the problematic map, datasources, dataWriteformats. This bug was caused by processing the SQLPlanInfo as they start/updated-by-the-AQE