NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
56 stars 37 forks source link

Add support for Photon-specific SQL Metrics #1390

Closed parthosa closed 4 weeks ago

parthosa commented 1 month ago

Fixes #1388

This PR adds support for Photon-specific SQL metrics as:

  1. cumulative time metric that can be used as a replacement for the scan time metric.
  2. shuffle write time can be reconstructed using the following metrics:
    1. time taken waiting on file write IO (part of shuffle file write)
    2. time taken to sort rows by partition ID (part of shuffle file write)
    3. time taken to convert columns to rows (part of shuffle file write)
  3. peak memory usage metric can be used for the peak execution memory metric.

Code Changes

Tests: