Open potiuk opened 1 year ago
I don’t think there’s a strict technical limitation, but a user perception considertation: Not all fields can reference an XComArg (e.g. outlets
and executor_config
can’t), so even if the limitation is lifted, it still wouldn’t be all fields. Instead of trying to teach users what fields can reference dynamic values and what can’t, it’s easier to positively list what can, and template_fields
is an easy one to go with.
Following the comment - I have a bold proposal ... It's not exaclty what the original proposal is, but in a way it provides a possibility to do what was originally requested here.
Why don't we add an option (disabled by default) to make ALL ELIGIBLE fields - "templated_fields" (and automatically .output -capable).
That bothered me for a while but I think there is very little impact of making all fields templated and often people complained that templated fields. Performance overhead should be negligible (just walking through parameters and jinjafying them which in most cases will be no-op).
The only drawback it might have is that the if a string contains " {{}}" acidentally - this will be replaced with "" - which is backwards-incompatible. We could also provide a mechanism that would eclude a field from being templated just in case.
I think that has a number of benefits - for example our users will not have extend operators that miss some fields in "templated_fields".
I am not too worried anout "outlets" and executor_config not being available for .output and user's education. As long as we simply error out in this case that should be good.
In a way it woudl be similar to render_template_as_native_obj
DAG paraemeter.
@potiuk Is this bold proposal still something you want to move forward with? I think it sounds interesting and I see how it can be useful to the Users. I can make an attempt at implementing the solution you have proposed.
Sure. Why not. I have not seen an opposition so far, and I think it's worth doing - I also see no big drawback if it's going to be similar setting to render_templates_as_native_obj
that you set for the whole DAG.
Hey Jarek, I've opened a PR for this. I took a slightly different approach than a DAG level setting. I managed the functionality at the Operator level, but this can still be applied to the whole DAG with the use of default_args
. I think that this approach gives the Users more flexibility. Looking forward to getting some feedback!
Hey Jarek, I've opened a PR for this. I took a slightly different approach than a DAG level setting. I managed the functionality at the Operator level, but this can still be applied to the whole DAG with the use of
default_args
. I think that this approach gives the Users more flexibility. Looking forward to getting some feedback!
Very nice. I asked others for input, but I like the way you proposed.
Discussed in https://github.com/apache/airflow/discussions/26938