Add qualification support for Photon jobs in the Python Tool - Githubissues

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

56 stars 37 forks source link

Add qualification support for Photon jobs in the Python Tool #1403

Closed parthosa closed 3 weeks ago

parthosa commented 3 weeks ago

Issue #251

This PR adds support for qualifying Photon jobs in the Python tool. We use a different strategy from Spark for qualifying Photon jobs.

For Spark jobs, recommend apps having speed up > 1.3x
For Photon jobs, recommend apps having speed up > 1x

Changes

Speed Up Strategy Builder

user_tools/src/spark_rapids_pytools/rapids/qualification.py:
- Introduced SpeedupStrategyBuilder and integrated it to dynamically build speedup strategies based on application type.
user_tools/src/spark_rapids_tools/tools/speedup_category.py:
- Added SpeedupStrategy and SpeedupStrategyBuilder classes to encapsulate speedup strategy logic.
- Modified SpeedupCategory to use these new classes. [1] [2] [3]

New Configuration Options:

user_tools/src/spark_rapids_pytools/resources/qualification-conf.yaml:
- Added metricsSubFolder and new speedup strategy configurations for different application types. [1] [2] [3]
user_tools/src/spark_rapids_pytools/rapids/tool_ctxt.py:
- Added get_metrics_output_folder method to retrieve the metrics output folder path.

Minor Code Adjustments:

.pylintrc: Removed redundant pylint disable rules.