aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

New EMR Serverless with SFN example #55

Closed meniluca closed 1 year ago

meniluca commented 1 year ago

Description of changes: adding a new example to integrate EMR Serverless (a)synchronized jobs submission with Step Function State Machines.

The code creates two example State Machines to submit EMR Serverless jobs asynchronously and synchronously.

Example: image image

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

meniluca commented 1 year ago

Changed the example to add the new .sync start job run now available in Step Functions.

meniluca commented 1 year ago

I should have covered everything 😃👍 , let me know if you would like to have all the commits squashed again. I have also removed a deprecated sfn function and repartitioned the Parquet write example to 1 not to generate too many files.

Have a good rest of the day! 👋

dacort commented 1 year ago

Thank you @meniluca !