search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
44 stars 34 forks source link

[BUG] Handle Missing CSP CLIs to Skip Cluster Inference #1034

Closed parthosa closed 1 month ago

parthosa commented 1 month ago

Currently, the cluster inference step attempts to create a CPU cluster based on the cluster shape. This requires the installation and configuration of CSP CLIs (aws/databricks). However, if the CLIs are not installed or configured, the tools can crash.

Proposed Solution:

  1. Log an appropriate error message.
  2. Skip the cluster inference process to prevent crashes.
  3. Disable cost savings associated with cluster inference when skipping it.