NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
56 stars 37 forks source link

[TASK] Add Arguments for Distributed Mode in Qualification Tool CLI #1430

Open parthosa opened 4 days ago

parthosa commented 4 days ago

To enable distributed execution in the Qualification Tool CLI, we need to begin by making updates to the current CLI to support this mode. This is required to set the stage for future implementation PRs

Changes

  1. Add Arguments to toggle between local and distributed execution modes.
  2. Provide a way to specify custom Spark properties that will be used to initialize the Spark Session.

Note:

Example configuration:

api_version: '1.0'
distributed_tools:
  spark_properties:
    - name: 'spark.executor.memory'
      value: '20g'