search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.27k stars 13.78k forks source link

Support network tags in airflow.providers.google.cloud.operators.dataproc.DataprocCreateBatchOperator() #40232

Open githubwua opened 2 weeks ago

githubwua commented 2 weeks ago

Description

When submitting a batch job to Dataproc, it is possible to specify network tags, as shown here

But airflow.providers.google.cloud.operators.dataproc.DataprocCreateBatchOperator() does not have an argument to accept network tags. So it is impossible to specify network tags when submitting Dataproc batch jobs from Airflow.

This is a feature request to add support for network tags in DataprocCreateBatchOperator()

Use case/motivation

When a network has firewall rules that allow/deny network traffic, we need to specify network tags to apply specific firewall rules to worker and master nodes. Without network tags, nodes cannot communicate with each other.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 2 weeks ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

kunaljubce commented 2 weeks ago

I want to take a stab at this, can you please assign this to me?

potiuk commented 2 weeks ago

Assigned.