search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
44 stars 34 forks source link

[BUG] Prediction mode should return speedup 1.0 instead of FallingBack to legacy Speedups #1058

Closed amahussein closed 1 month ago

amahussein commented 1 month ago

Describe the bug There are some cases where the QualX prediction would fall back to the legacy speedup. Instead the QualX prediction should set the speedup to 1.0

case-1

In this case, all stages are unsupported for this sqlID. The logic should be that the sqlID gets a speedup of 1

https://github.com/NVIDIA/spark-rapids-tools/blob/270763d036248d6a92cd86ac740863c3[…]19291d/user_tools/src/spark_rapids_tools/tools/model_xgboost.py

    if node_level_supp is not None and (qualtool_filter == 'stage'):
        # if supported exec info supplied aggregate features only over supported stages
        sql_job_agg_tbl = job_stage_agg_tbl.loc[job_stage_agg_tbl['Exec Is Supported']]
        if sql_job_agg_tbl.empty:
            logger.warning('No fully supported stages.')

TODO: enumerate other cases