search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
49 stars 36 forks source link

[BUG] user tools Qualification tunings and node recommendation when --cluster specified can be wrong #1147

Open tgravescs opened 3 months ago

tgravescs commented 3 months ago

Describe the bug The current behavior of the spark_rapids qualification tool is that if the user specifies the --cluster property that is what was used to run all the applications being qualified. This is a huge assumption and could cause the tunings and qualification to be wrong.

The only time this actually works is if all the apps being qualified were run on the exact same type of cluster individually.

I think we should do something about this to make sure its not giving confusing results.

amahussein commented 3 months ago

We need to decide what we need to do moving forward. This could be part of the change needed to support cost-savings per app (instead of batch)

CC: @mattahrens