[BUG] Cluster recommendation for executors with varying number of cores

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

53 stars 37 forks source link

[BUG] Cluster recommendation for executors with varying number of cores #1255

Open parthosa opened 3 months ago

parthosa commented 3 months ago

There could be a corner case when an app was run on a cluster having varying number spark executor cores. We should be able to handle this case - correctly identify the source cluster shape and estimate the recommended cluster shape.

Example

A Spark App having executors with different number of cores: 96, 96, 16, 80, and 96

cc: @tgravescs

tgravescs commented 3 months ago

Note we saw this happen on an EMR cluster running Spark 2.x. So this was on yarn and I have never seen executors have different number of cores on YARN.