NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
50 stars 37 forks source link

Update prediction code to use separate job and stage level aggregates #1047

Closed parthosa closed 4 months ago

parthosa commented 4 months ago

Issue: #1017

1044 added support to split job and stage level aggregates in two separate files.

Based on offline discussion with @eordentlich, this PR updates the prediction code to concatenate the separate job and stage level aggregates so that they can be used. This is to ensure minimal changes in the existing code.

We will refactor this once we have updated the code base with the latest changes from @leewyang.