NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
56 stars 38 forks source link

Reduce default memory allocation to the java process #1407

Closed amahussein closed 3 weeks ago

amahussein commented 3 weeks ago

Fixes #1406

This pull request introduces several changes to improve the handling of JVM heap size and thread calculations in the spark_rapids_tools module. The most important changes include updating the method for calculating JVM heap size.

This change aims at avoiding allocating memory by default that would trigger the OOM-killer

Enhancements to JVM heap size and thread calculations:

Method renaming for clarity: