Add default fallback for unsupported platform and spark runtime configurations - Githubissues

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

56 stars 37 forks source link

Add default fallback for unsupported platform and spark runtime configurations #1421

Open parthosa opened 1 week ago

parthosa commented 1 week ago

Fixes #1420.

This PR adds features to handle cases where the user provides an event log with a runtime not supported by the current platform.

Example

A user provides an event log with a runtime (e.g., photon) that is incompatible with the specified platform (e.g., onprem).

In that case, the JAR tool will set the runtime to the platform’s default and generate a warning as (word-wrapped for readability):

WARN QualificationAppInfo: Application app-xxxx-0000: \
Platform 'onprem' does not support the parsed runtime 'PHOTON'. \
 Falling back to default runtime - 'SPARK'.

Changes

Enhancements to Platform and Runtime Handling:

Added defaultRuntime and supportedRuntimes properties to the Platform class, along with a method to check if a runtime is supported (core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala).
Updated DatabricksPlatform to include PHOTON in its supported runtimes (core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala).

Refactorings in Profiling Tool:

Modified Profiler class to initialize a Platform instance and pass it to ApplicationInfo (core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala). [1] [2]
Updated ApplicationInfo to accept a Platform parameter and use it in the base class (AppBase) (core/src/main/scala/org/apache/spark/sql/rapids/tool/profiling/ApplicationInfo.scala).

Adjustments in AppBase Class:

Added platform parameter and logic to handle unsupported runtimes gracefully, falling back to the platform’s default runtime (core/src/main/scala/org/apache/spark/sql/rapids/tool/AppBase.scala). [1] [2]

Tests

Included unit tests to validate runtime detection and fallbacks for different platforms (core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/ApplicationInfoSuite.scala).