[Feature] - Use AWS serverless workflow service step functions from EKS

[victorgu-github]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

[AWS Step Functions] which is a serverless workflow service can integrate with Amazon EMR on EKS and [Amazon EventBridge] to build event-driven workflows. After installing [AWS Controllers for Kubernetes (ACK) in EKS, you can provision and configure serverless AWS resources: Amazon EventBridge and AWS Step Functions from EKS. The team can do the whole data operation without leaving the Kubernetes platform and only need to maintain the EKS cluster since all the other components are serverless.

The airflow infra is too heavy for some users. An data pipeline with serverless services will offload a lot from admin works while ACK controllers allow them to stay in EKS to control and config those serverless services.

Describe the solution you would like

Describe alternatives you have considered

airflow, argo workflows

Additional context

[victorgu-github]

Based on the comments from Vara and Apoorva, replace ACK and just use terraform

[vara-bonthu]

Lets push this feature to EMR on EKS Workshop content

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

Issue closed due to inactivity.