[BUG] Pinned pool size setting may be missing from the autotuner recommendation

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

56 stars 37 forks source link

[BUG] Pinned pool size setting may be missing from the autotuner recommendation #1398

Open kuhushukla opened 3 weeks ago

kuhushukla commented 3 weeks ago

Describe the bug We do not always recommend the user to have a pinned pool size setting with an appropriate overhead memory setting. spark.rapids.memory.pinnedPool.size . This was seen in on-prem environments.

We should verify if this setting is set appropriately with respect to other params as well. eg. spark.executor.memory and spark.executor.memoryOverhead