search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.86k stars 14.24k forks source link

Would it be possible to add 'config' to the list of template fields for EmrServerlessStartJobOperator? #31711

Closed sumitdas66 closed 1 year ago

sumitdas66 commented 1 year ago

Description

We have a use case where we would like to start job runs in EMR Serverless where the job name is derived from the output from a previous task. Since the job name is passed via the 'config' argument to the EmrServerlessStartJobOperator, this request is to add 'config' to the list of 'template_fields' in that operator (https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/emr.py#L966), if at all possible.

Thanks in advance, Sumit Das

Use case/motivation

No response

Related issues

Something similar is already implemented on the aws-samples project. Link to the related issue: https://github.com/aws-samples/emr-serverless-samples/issues/36 Link to the code: https://github.com/aws-samples/emr-serverless-samples/blob/main/airflow/emr_serverless/operators/emr.py#L171

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

phanikumv commented 1 year ago

This should definitely be feasible! Let me know if you would like to open a PR on it ?

mahammi commented 1 year ago

Hello! I would like to pick up this issue.

o-nikolas commented 1 year ago

Assigned to you @mahammi :)

sumitdas66 commented 1 year ago

Which versions of the provider and airflow can we expect these changes to reflect?

potiuk commented 1 year ago

Which versions of the provider and airflow can we expect these changes to reflect?

Observe release notes and announcement on users@ when it will be released. Checking release notes is the easiest way to verify if the PR fixes it.

phanikumv commented 1 year ago

Hi @potiuk , I believe a PR needs to be created for the issue, hence lets keep the issue open.

o-nikolas commented 1 year ago

Hi @potiuk , I believe a PR needs to be created for the issue, hence lets keep the issue open.

@phanikumv you can see the PR from Maham above has been merged (it is #31746), so I think this is safe to close