Closed tgravescs closed 3 weeks ago
Here are a few thoughts on a couple of the CSPs to get discussions going:
open questions on naming - use "worker" or "executor"? Or does it change based on csp?
Note you might have multiple executors per node but I think that info can live in the tunings files.
Databricks:
"platform": "databricks-aws",
"driverNodeType": "g5.2xlarge",
"workerNodeType": "g5.8xlarge",
"numWorkerNodes": 2,
"gpuInfo": {
# here gpu type is decided by worker node type so don't need... but might be useful to specify if we know it?
"gpusPerWorker": 1
}
Dataproc:
"platform": "dataproc",
"driverNodeType": "n1-standard-4",
"workerNodeType": "n1-standard-32",
"numWorkerNodes": 2,
"gpuInfo": {
"device": "nvidia-tesla-t4",
"gpusPerWorker": 2
},
"additionalConfigs": {
"localSsds": 2
}
Dataproc serverless: TODO
Dataproc GKE: TODO
EMR: TODO
onPrem:
"platform": "onprem",
"numWorkerNodes": 2, ? do we even want this? There is a difference between standalone and yarn mode so this could still be useful. For yarn we also want to make sure node for the driver to go on
"gpuInfo": {
"gpusPerWorker": 2
},
Another thing to revisit is the name of this file "qualification_summary_metadata.json". Perhaps app_metadata would be more obvious to the user..
Also similar as above output figure out what we want the SourceCluster output to be. Generally I would expect it to match the recommendedCluster output
Thanks @tgravescs for the comments.
open questions on naming - use "worker" or "executor"? Or does it change based on csp?
- Dataproc, DB-AWS and DB-Azure use the
worker
terminology while creating clusters. EMR usescore/task
. We should probably useworker/workerNode/workerNodeType
Another thing to revisit is the name of this file "qualification_summary_metadata.json". Perhaps app_metadata would be more obvious to the user..
app_metadata.json
makes senseNote you might have multiple executors per node but I think that info can live in the tunings files.
- Yes, metadata file should contain entries related to cluster shape. If we distinguish 'workers' and 'executors', then all
executor
related info could reside in the tuning files.
Currently here are the cluster info generated in the metadata json file (ps: similar structure for source cluster)
"recommendedCluster": {
"driverInstance": "n1-standard-16",
"executorInstance": "n1-standard-32",
"numExecutors": 4,
"gpuInfo": {
"device": "nvidia-tesla-t4",
"gpuPerWorker": 1
},
"additionalConfig": {
"localSsd": 2
}
}
"recommendedCluster": {
"driverInstance": "m6gd.xlarge",
"executorInstance": "g5.2xlarge",
"numExecutors": 2
}
"recommendedCluster": {
"driverInstance": "Standard_E8ds_v4",
"executorInstance": "Standard_NC8as_T4_v3",
"numExecutors": 2
}
"recommendedCluster": {
"driverInstance": "i3.2xlarge",
"executorInstance": "g5.4xlarge",
"numExecutors": 16
}
"recommendedCluster": {}
For instance on Databricks the executor instance type might be enough and we don't need the number of gpus, but we would want a number of worker nodes, not the number of executors.
- For Databricks we are showing only the executor instance type.
numExecutors
refer to number of worker nodes (we should rename this if we decide to move withworker
)GCP we may want to see a number of gpus and number of local ssds to use.
- We show the
gpuPerWorker
andlocalSsd
to use (though its value is hard coded to '2')But on prem could be different since we don't know the executor instance types available and generally just use number of executors.
- We need to decide the cluster recommendation for OnPrem
platform
, we could probably have the structure as:
"clusterInfo": {
"platform": "databricks-aws",
"sourceCluster": {
"driverInstance": "m6gd.xlarge",
"executorInstance": "m6gd.2xlarge",
"numExecutors": 2
},
"recommendedCluster": {
"driverInstance": "m6gd.xlarge",
"executorInstance": "g5.2xlarge",
"numExecutors": 2
}
},
Agree with the platform at the clusterInfo level.
I would lean towards workerNodeType, but we may also want to make this consistent. I think the rapids_4_spark_qualification_output_cluster_information.json uses numExecutorNodes. Can you check if its referred to anywhere else in actual user visible output? I'm not as concerned about it matching terminology CSPs use as its concistent in our usage.
For on prem, I think we make some assumptions on number of gpus per node and number of nodes so we should output what those are and fill in the recommendedCluster. We need to decide if we want the executorInstance to just not be there or say something like Not applicable. Its sometimes nice to always have some output there.
Can you check if its referred to anywhere else in actual user visible output?
- I checked the output files. We refer to
numExecutors
only in therapids_4_spark_qualification_output_cluster_information
and the metadata file.- I think the following structure looks reasonable
"recommendedCluster": { "driverNodeType": "m6gd.xlarge", "workerNodeType": "g5.8xlarge", "numWorkerNodes": 2 }
- Now,
executor
term will always refer to spark executors- For OnPrem, we can have the following structure:
"recommendedCluster": { "driverNodeType": "Not Applicable", "workerNodeType": "Not Applicable", "numWorkerNodes": 2, "gpuInfo": { "device": "nvidia-tesla-t4", "gpuPerWorker": 1 } }
Its sometimes nice to always have some output there.
- I agree. Fields
driverNodeType
,workerNodeType
andnumWorkerNodes
should be present for all platforms.
onprem
in the metadata filegpuPerWorker
is recommended for eligible platforms (dataproc
, onprem
) worker
prefix in the output files (i.e. workerNodeType
and numWorkerNodes
)app_metadata.json
and add platform
field@tgravescs Does the above list look good?
looks good, thanks!
For the follow up, I plan to improve the cluster recommendation for OnPrem and Dataproc as follows:
File: qual_2024xxx/app_metadata.json
Qualified Cluster Recommendation
column
PR https://github.com/NVIDIA/spark-rapids-tools/pull/1216 (came from https://github.com/NVIDIA/spark-rapids-tools/issues/1143) added a qualification_summary_metadata.json file. It has a recommendedCluster field in there that I think could enhanced to be more clear to the user.
We need to talk though all the CSPs/cluster types and decide what we want that output to be.
I think it should include the platform in the metadata file as well. For instance on Databricks the executor instance type might be enough and we don't need the number of gpus, but we would want a number of worker nodes, not the number of executors. But on prem could be different since we don't know the executor instance types available and generally just use number of executors. GCP we may want to see a number of gpus and number of local ssds to use.