NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
44 stars 34 forks source link

[FEA] Enhance qualification tool Auto Tuner CPU event log recommendations #1068

Closed tgravescs closed 4 weeks ago

tgravescs commented 1 month ago

Is your feature request related to a problem? Please describe. We hooked the basic metrics up to the auto tuner when running with the qualification tool. Now we need to enhance/fix the heuristics to apply to the CPU event log. There were also some assumptions being made about memory sizes. Investigate adding the number of shuffle and reader threads settings based on node configurations.