search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
150 stars 74 forks source link

Would it be possible to add 'config' to the list of template fields for EmrServerlessStartJobOperator? #36

Closed fventuri-availity closed 1 year ago

fventuri-availity commented 2 years ago

We have a use case where we would like to start job runs in EMR Serverless where the job name is derived from the output from a previous task. Since the job name is passed via the 'config' argument to the EmrServerlessStartJobOperator, this request is to add 'config' to the list of 'template_fields' in that operator (https://github.com/aws-samples/emr-serverless-samples/blob/main/airflow/emr_serverless/operators/emr.py#L144-L149), if at all possible.

Thanks in advance, Franco Venturi

dacort commented 1 year ago

Hey Franco - this should definitely be feasible! There will likely be a new release in the coming weeks.