search

awslabs / idf-modules

Industry Data Framework (IDF) IAC modules repository
Apache License 2.0
21 stars 12 forks source link

[FEATURE] Spark on EKS #200

Closed kevinsoucy closed 2 days ago

kevinsoucy commented 6 days ago

Is your feature request related to a problem? Please describe. ADAS customers are looking for a way kubernetes-native way to run Spark Applications, using Karpenter for autoscaling. We would like a demo-able Step Function using the EKS module to spark submit an application to K8s

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

srinivasreddych commented 2 days ago

EMR on EKS works in Localzones. Tested and linked the PR