search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
49 stars 36 forks source link

[FEA]Qualification tool to qualify a whole dataproc serverless DAG instead of a single Spark Application #1090

Open viadea opened 3 months ago

viadea commented 3 months ago

Is your feature request related to a problem? Please describe. I wish Qualification tool can qualify a whole dataproc serverless DAG instead of a single Spark Application

Describe the solution you'd like Say there are 10 Spark Applications(dataproc serverless batch jobs) running in the same DAG. Users need to decide if they want to migrate the whole DAG from CPU to GPU. The qualification tool should look at all 10 Spark applications to make a DAG-level decision to see if this DAG is qualified or not.

amahussein commented 3 months ago

Waiting for Felix to decide the requirements