[FEA] Auto tuning/node recommendation missing features

[tgravescs]

Is your feature request related to a problem? Please describe. This is a list of things I ran into that I think is missing or could make the users life easier:

• [x] 1. Databricks - specify the number of workers since it doesn't specify spark.executor.instances
• [ ] 2. Databricks - make sure we handle the case with multiple executors per node
• [ ] 3. EMR - expand recommendation logic to handle multiple GPU nodes
• [ ] 4. EMR - need to specify the number of workers as well. specifically in the case we go to machines with 1 GPU and cpu could have been using larger nodes with less of them.
• [ ] 5. . Some of our recommendations combine executors to choose machine size but then the ones we have only have a single GPU so we are halving the number of executors, which with GPU could have a big performance impact.. for instance CPU side on EMR has 16 cores and using 2 executors per node. We will say to use nodes with 32 cores but it only has 1 GPU.
• [ ] 6. In recommending the number of nodes currently we will recommend more if you have to have more nodes because smaller number of gpus supported to get the same number of executors, but we don't handle the opposite case where maybe we recommend a node with more GPUs per node so can put more executors on it and so would need less nodes.
• [x] 7. Node types shouldn't be hardcoded- should be an issue for this already so marking done
• [x] 8. More Azure node types
• [ ] 9. Add driver instance recommendation to scala code
• [x] 10. enhance python shape recommendation to include num gpus and num nodes - https://github.com/NVIDIA/spark-rapids-tools/issues/1143 - do it under linked issue
• [x] 11. dataproc auth error is not obvious enough when node recommendation fails - https://github.com/NVIDIA/spark-rapids-tools/issues/1126
• [x] 12. switch hardcoded node maps to be lookup from json files generated from node types on each CSP - duplicate of item 7
• [x] 13. replace hardcoded instance information in Platform.scala with files generated from CSP information
• [ ] 14. Databricks-azure, pass in an databricks aws event log with platform as databricks-azure and no cluster and the final output on python doesn't choose the correct node - it comes out nan whereas scala figured out proper thing.
• [ ] 15. revisit on prem defaults.. if >=16 cores defaults to 2 gpus... this does work when mapping to dataproc but we don't print out create command
• [x] 16. Do we want to deal with local mode for resources? ignore for now
• [ ] 17. For k8s overhead memory we are assuming its JVM based.. k8s actually uses different overhead factors if python vs jvm based.
• [ ] 18. Cluster properties file (worker_info.yaml) is loaded twice in many cases for autotuner for qual and profiling tool.
• [ ] 19. Handle yarn cluster node recommendations better. yarn could have different number of executors per node. see if we can be more certain about usage or figure out how to display this better to the user.

[tgravescs]

this needs to be broken up @tgravescs to handle that once list is finalized

[tgravescs]

marking items that have other issues as done in this one. I may leave this open as many of these are smaller and might be able to be done together.