To enable distributed execution in the Qualification Tool CLI, we need to begin by making updates to the current CLI to support this mode. This is required to set the stage for future implementation PRs
Changes
Add Arguments to toggle between local and distributed execution modes.
Provide a way to specify custom Spark properties that will be used to initialize the Spark Session.
Note:
For specifying custom Spark properties, instead of introducing a new argument, we will utilize the existing --tools_config_file option.
To enable distributed execution in the Qualification Tool CLI, we need to begin by making updates to the current CLI to support this mode. This is required to set the stage for future implementation PRs
Changes
Note:
-tools_config_file
option.Example configuration: