Open kimminw00 opened 9 months ago
This might be blocked by https://github.com/apache/airflow/issues/29069 .
In general you could extend S3CreateObjectOperator.template_fields
by create custom Operator with required template_fields
from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator
class AwesomeS3CreateObjectOperator:
template_fields: ("aws_conn_id", *S3CreateObjectOperator.template_fields)
I think it might be a good first issue, I also required to collect all connection ID from the existed operators and list them, so everyone could pick up and make the changes.
Mark as good first issue, so maybe someone could volunteer free time to find at least most of the non-templated connections ids.
I collected all third party operators which have connection IDs.
I think we would just add the connection_id paramter to template_fields
for all existing providers operators? However, that would be cumbersome and how to apply that convention moving forward? I'd be happy to make the updates to the operators, but I'm concerned about inconsistency.
Description
We use private staging and prod S3s(Ceph clusters for example) in our office. So there are often cases where DAGs are running with only connection ids changed. We prefer to use
Param
rather than to use hardcoded connection ids to make our code reusable. I only gave an example for Amazon operator, but templating connection ids is required for other operators too.Why is it needed? Code reusability
Use case/motivation
Related issues
No response
Are you willing to submit a PR?
Code of Conduct