Add internal CLI to generate instance descriptions for CSPs

[cindyyuanjiang]

Fixes https://github.com/NVIDIA/spark-rapids-tools/issues/1123.

This PR is first step to remove dependency on CSP CLIs. Ideally after we added this CLI, we can generate instance description json files for each platform and store them as resources in tools repo. Then we can add logic to use these instance description files when running tools.

Changes

Added an internal CLI spark_rapids_dev generate_instance_description [options].

Options:

• platform: currently accepts emr, dataproc and databricks-azure. Databricks-aws platform can reuse the file generated by emr, and dataproc-gke can reuse the one for dataproc.
• output_folder: directory path for the output file.

The generated json file has the following format (which is inspired by EMR CLI output):

{
  "instance_name": {
     "VCpuInfo": {
       "DefaultVCpus": xxx
     },
     "MemoryInfo": {
       "SizeInMiB": xxx
     },
     "GpuInfo": {
       "Gpus": [
         {
           "Name": "xxx",
           "Manufacturer": "xxx",
           "Count": xxx,
           "MemoryInfo": {
             "SizeInMiB": xxx
           }
         }
       ]
     }
   },
  ...
}

For CPU instance, the entry for "GpuInfo" will look like "GpuInfo": {}.

Example json file entry for EMR platform

"g5.4xlarge": {
    "VCpuInfo": {
      "DefaultVCpus": 16
    },
    "MemoryInfo": {
      "SizeInMiB": 65536
    },
    "GpuInfo": {
      "Gpus": [
        {
          "Name": "A10G",
          "Manufacturer": "NVIDIA",
          "Count": 1,
          "MemoryInfo": {
            "SizeInMiB": 24576
          }
        }
      ]
    }
  }

Testing

spark_rapids_dev generate_instance_description --platform emr/dataproc/databricks-azure

[cindyyuanjiang]

I want to discuss: is it a good approach to add a new spark_rapids_dev CLI or should we keep it under spark_rapids?

[mattahrens]

I would rather keep it under the spark_rapids CLI instead of adding a new one. Is there a way to add the command without exposing it to users since it should be internal?

[tgravescs]

what would be the goal of keeping it under the same cli an end user would use without having a useful info message for anyone including dev to see?

[amahussein]

I would rather keep it under the spark_rapids CLI instead of adding a new one. Is there a way to add the command without exposing it to users since it should be internal?

@mattahrens In our CLI, it is not possible to hide a cmd/argument.

[amahussein]

I want to discuss: is it a good approach to add a new spark_rapids_dev CLI or should we keep it under spark_rapids?

Thanks @cindyyuanjiang ! I will take a look at the changes.

[mattahrens]

I would rather keep it under the spark_rapids CLI instead of adding a new one. Is there a way to add the command without exposing it to users since it should be internal?

@mattahrens In our CLI, it is not possible to hide a cmd/argument.

OK, then we can have a dev CLI then for separation.