[FEA] AutoTuner investigate configuring spark.executor.cores to an optimal ratio per GPU

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

44 stars 34 forks source link

[FEA] AutoTuner investigate configuring spark.executor.cores to an optimal ratio per GPU #1067

Open tgravescs opened 1 month ago

tgravescs commented 1 month ago

Is your feature request related to a problem? Please describe. Investigate if the auto tuner could recommend a ratio of spark.executor.cores to GPU to make performance and TCO the best. Perhaps look at semaphore wait time or other metrics.