search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
43 stars 34 forks source link

[BUG] Cluster Information: Include number of executors per node #1117

Closed parthosa closed 2 weeks ago

parthosa commented 2 weeks ago

Currently for cluster information deduction, we calculate the number of nodes correctly but do not track the number of executors per node.

This can generate wrong GPU cluster recommendations because there can be multiple executors per node. We should track number of executors per node.

tgravescs commented 2 weeks ago

Note this is basically a subtask of fixing https://github.com/NVIDIA/spark-rapids-tools/issues/1109