search

awslabs / data-on-eks

DoEKS is a tool to build, deploy and scale Data & ML Platforms on Amazon EKS
https://awslabs.github.io/data-on-eks/
Apache License 2.0
653 stars 221 forks source link

[Feature] EMR on EKS with Volcano Scheduler #12

Open vara-bonthu opened 2 years ago

vara-bonthu commented 2 years ago

Part1 of the PR

1/ Add this add-on deployment to internal TF modules . Here https://github.com/awslabs/data-on-eks/tree/main/workshop/modules/terraform-aws-eks-data-addons

2/ Add this add-on to emr-eks-karpenter pattern with a create_volcano variable and set it to false as default. Users will enable either Volcano or YuniKorn but not both

3/ Add an example under https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/emr-eks-karpenter/examples/nvme-ssd to show Volcano with gang scheduling

4/ Update the Website Docs to explain the execution process and the results

Part2 of the PR

1/ Add this add-on to spark-k8s-operator pattern with a create_volcano variable and set it to false as default. Users will enable either Volcano or YuniKorn but not both

2/ Add an example under https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/spark-k8s-operator/examples/karpenter to show Volcano with gang scheduling

3/ Update the Website Docs to explain the execution process and the results

Use this example as a template and build on top of that. https://github.com/awslabs/data-on-eks/tree/main/analytics/emr-eks-amp-amg

vara-bonthu commented 1 year ago

No new blueprint is required. Just add a Volcano Helm chart to the existing blueprint https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/emr-eks-karpenter and Spark examples to run the jobs

vara-bonthu commented 1 year ago

Let's create a Github workflow for Volcano to publish the Helm artifacts so that it can be deployed easily. See this issue https://github.com/volcano-sh/volcano/issues/2766