Add support for disk spill metrics in Photon jobs - Githubissues

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

56 stars 38 forks source link

Add support for disk spill metrics in Photon jobs #1405

Closed parthosa closed 3 weeks ago

parthosa commented 3 weeks ago

Issue #251

Following up on #1390, this PR adds parsing for Disk Spill Size metrics for Photon Jobs.

Photon event logs report disk spill size using the metric name "num bytes spilled to disk".
Similar to 1390, we now calculate the disk spill size using accumulators instead of task metrics for Photon jobs.
Note: Could not find any metrics for in-memory spills in Photon event logs.

Changes

Enhancements to Photon-specific metrics handling:

core/src/main/scala/com/nvidia/spark/rapids/tool/analysis/AppSparkMetricsAnalyzer.scala: Use a HashMap to store Metric -> Accumulator and added support for disk spill size metrics. [1] [2]
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/DatabricksParseHelper.scala: Added a new label for disk spill size metrics.

Test expectation updates:

core/src/test/resources/ProfilingExpectations/nds_q88_photon_db_13_3_job_metrics_agg_expectation.csv: Updated to include disk spill size metrics.
core/src/test/resources/ProfilingExpectations/nds_q88_photon_db_13_3_sql_metrics_agg_expectation.csv: Updated to include disk spill size metrics.
core/src/test/resources/ProfilingExpectations/nds_q88_photon_db_13_3_stage_metrics_agg_expectation.csv: Updated to include disk spill size metrics.
Updated event log nds_q88_photon_db_13_3.zstd to include spill metrics

parthosa commented 3 weeks ago

This PR is no more required since spill metrics are parsed correctly by default.