This pull request introduces several changes to improve the handling of JVM heap size and thread calculations in the spark_rapids_tools module. The most important changes include updating the method for calculating JVM heap size.
This change aims at avoiding allocating memory by default that would trigger the OOM-killer
use available memory instead of total.
cap the xmx to 32 GB
cap the max number of threads to 8
Enhancements to JVM heap size and thread calculations:
user_tools/src/spark_rapids_tools/cmdli/argprocessor.py: Updated the method to calculate JVM heap size using Utilities.calculate_jvm_max_heap_in_gb() instead of Utilities.get_system_memory_in_gb(). Increased the minimum heap size per thread from 6 GB to 8 GB.
user_tools/src/spark_rapids_tools/utils/util.py: Renamed get_system_memory_in_gb() to calculate_jvm_max_heap_in_gb() and updated the method to calculate the maximum heap size based on available system memory, capping it between 8 GB and 32 GB.
user_tools/src/spark_rapids_tools/utils/util.py: Renamed get_max_jvm_threads() to calculate_max_tools_threads() and updated the method to calculate the maximum number of threads based on physical cores, capping it at 8 threads.
user_tools/src/spark_rapids_tools/utils/util.py: Updated the adjust_tools_resources method to use the new calculate_max_tools_threads() method for determining the maximum number of threads.
Fixes #1406
This pull request introduces several changes to improve the handling of JVM heap size and thread calculations in the
spark_rapids_tools
module. The most important changes include updating the method for calculating JVM heap size.This change aims at avoiding allocating memory by default that would trigger the OOM-killer
Enhancements to JVM heap size and thread calculations:
user_tools/src/spark_rapids_tools/cmdli/argprocessor.py
: Updated the method to calculate JVM heap size usingUtilities.calculate_jvm_max_heap_in_gb()
instead ofUtilities.get_system_memory_in_gb()
. Increased the minimum heap size per thread from 6 GB to 8 GB.user_tools/src/spark_rapids_tools/utils/util.py
: Added class variablesmin_jvm_xmx
,max_jvm_xmx
, andmax_tools_threads
to set limits on JVM heap size and the number of threads.Method renaming for clarity:
user_tools/src/spark_rapids_tools/utils/util.py
: Renamedget_system_memory_in_gb()
tocalculate_jvm_max_heap_in_gb()
and updated the method to calculate the maximum heap size based on available system memory, capping it between 8 GB and 32 GB.user_tools/src/spark_rapids_tools/utils/util.py
: Renamedget_max_jvm_threads()
tocalculate_max_tools_threads()
and updated the method to calculate the maximum number of threads based on physical cores, capping it at 8 threads.user_tools/src/spark_rapids_tools/utils/util.py
: Updated theadjust_tools_resources
method to use the newcalculate_max_tools_threads()
method for determining the maximum number of threads.