[FEA] Support for Dataproc Serverless platform in spark-rapids-user-tools

[SurajAralihalli]

Adding support for the Dataproc Serverless platform in spark-rapids-user-tools would be useful to the Serverless users.

1. Qualification Tool
2. Profiling Tool

### Tasks
- [ ] https://github.com/NVIDIA/spark-rapids-tools/issues/663

[amahussein]

Do we mean by "support" the ability to run the spark-rapids-user-tools python package on Dataproc serverless? Or the ability to analyze the logs generated by apps running on serverless platform?

[mattahrens]

P0 scope:

• [ ] Add dataproc-serverless as a supported platform and validate that running qualification tool with this platform uses the available speedup factors for Dataproc Serverless.

P1 scope:

• [ ] Support dataproc-serverless cost estimation for the qualification tool

[SurajAralihalli]

Do we mean by "support" the ability to run the spark-rapids-user-tools python package on Dataproc serverless? Or the ability to analyze the logs generated by apps running on serverless platform?

Hello @amahussein, I meant the latter one. The ability to analyze the logs generated by Dataproc Serverless applications (qualification, profiling). On a similar note #663 is a feature request to add the Dataproc Serverless job creation command in the qual tool output for Dataproc (for users migrating from classic dataproc to serverless)

[amahussein]

After assessing this feature, we discussed that we need to be able to detect the cluster shape from the eventlogs. The reason is that users won't likely use the batchId of spark submission around for long. This means that we cannot use the batchID or cluster configs as inputs for the tools. Instead we have to rely on extracting the information from evntlogs/driverlogs.

So, this issue should depend on #581

Resources:

Spark properties for dataproc Serverless

https://cloud.google.com/dataproc-serverless/docs/concepts/properties

serverless pricing estimation:

https://cloud.google.com/dataproc-serverless/pricing