NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
50 stars 37 forks source link

Fix parsing App IDs inside metrics directory in QualX #1167

Closed parthosa closed 3 months ago

parthosa commented 3 months ago

Fixes #1163. QualX assumes App IDs start with app* and uses this regex pattern to fetch profiling metrics. However, App IDs can be any alphanumeric string.

This PR fixes the parsing to read all subdirectories and create the list. We are guaranteed that all sub directories within raw_metrics contain individual app metrics so we do not need a regex matching and can safely read all of them.