user-tools should add xms argument to java cmd

[amahussein]

Signed-off-by: Ahmed Hussein ahussein@nvidia.com

Fixes #1382

Upon investigation, it was revealed that the min heap size could impact the runtime significantly. (see the linked issue for details of the performance impact) This code change aims at setting the xms java argument to 50% of the max heap size.

• pass xms to the java cmd
• update the runtime report to list jvm info along with jvm arguments related to heap:
  • runtime.jvm.*
  • runtime.jvm.arg*

sample runtime.properties file. The lines that are generated by the changes in this PR are marked with >

#RAPIDS Accelerator for Apache Spark's Build/Runtime Information
#Wed Oct 23 20:26:21 UTC 2024
build.scala.version=2.12.15
build.hadoop.version=3.3.6
build.spark.version=3.5.0
> runtime.os.version=6.8.0-39-generic
> runtime.jvm.version=1.8.0_422
build.version=24.08.3-SNAPSHOT
runtime.spark.version=3.4.2
> runtime.jvm.arg.heap.min=50g
> runtime.jvm.name=OpenJDK 64-Bit Server VM
> runtime.jvm.arg.gc.UseG1GC=
> runtime.os.name=Linux
> runtime.jvm.arg.heap.max=100g
build.java.version=1.8.0_422

Details

This pull request updates the handling of JVM heap arguments in the Spark RAPIDS tool.

• The most important is setting the Xms the jar CLI.
• Introduces enhancements to the RuntimeUtil class by adding JVM and OS information to the runtime properties, extracting JVM heap arguments, and ensuring these arguments are correctly set in the user tools.

Enhancements to RuntimeUtil:

• core/src/main/scala/org/apache/spark/sql/rapids/tool/util/RuntimeUtil.scala: Added imports for ManagementFactory and Scala collection conversions.
• core/src/main/scala/org/apache/spark/sql/rapids/tool/util/RuntimeUtil.scala: Added methods to include JVM and OS information in the runtime properties and to extract JVM heap arguments. [1] [2]

Updates to JVM heap arguments handling:

• user_tools/src/spark_rapids_pytools/rapids/rapids_tool.py: Updated the _re_evaluate_platform_args method to set both minimum and maximum heap size arguments for the JVM.
• user_tools/src/spark_rapids_tools/utils/util.py: Added logic to calculate and include the minimum heap size in the tool resources.

Related and followups:

• internal ticket was filed to update the java CLI in the CI/CD job
• there is an internal ticket to update the user guide to emphasize the important of setting the heap arguments.

[nartal1]

Thanks @amahussein for investigating and putting up the fix for this. Just a nit.

[amahussein]

Thanks @amahussein for investigating and putting up the fix for this. Just a nit.

Thanks @nartal1 ! Removed the debugging code.