Closed yuqian90 closed 1 year ago
Note that for any operator, if it uses macros in user_defined_macros
in jinja template fileds and the user clicks on "Rendered Template" on the airflow page, they get an error message saying "Webserver does not have access to User-defined Macros or Filters". This is how the error is handled on the webserver for "Rendered Template". This is the exact same problem as the one reported in this issue. Both are because airflow Webserver do not have reference to the user_defined_macros
of the DAG. If we have plans for making "Rendered Template" work for user_defined_macros
, the same solution can probably be applied to this ExternalTaskMarker
issue reported here.
Or if there's no plan to solve the "Rendered Template" issue, maybe I can put up a PR to handle the ExternalTaskMarker
error in the same fashion (i.e. catch exception and flash an error message on the website when it happens).
@kaxil what do you think?
Tried to discuss this on slack in #airflow-2-0. One possible workaround at the moment is to use AirflowPlugin
instead of user_defined_macros
. AirflowPlugin
works fine in the webserver so jinja templates can be rendered just fine. The only downside is that macros are more cumbersome to use in jinja templates. E.g. with user_defined_macros
, we could do {{ next_holiday(execution_date) }}
. But with AirflowPlugin
, we have to write {{ macros.my_plugin.next_holiday(execution_date) }}
.
@kaxil suggested the following small change to dag.clear()
which should make ExternalTaskMarker
that have already run play well with dag.clear()
because get_rendered_template_fields()
reads from the db:
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index 15332f334..30aae0083 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -1260,7 +1260,7 @@ class DAG(LoggingMixin):
max_recursion_depth, ExternalTaskMarker.__name__, ti.task_id
)
)
- ti.render_templates()
+ ti.get_rendered_template_fields()
external_tis = session.query(TI).filter(
TI.dag_id == task.external_dag_id,
TI.task_id == task.external_task_id,
Hey @yuqian90 , did the proposed solution work for you locally? Do you think we should make a PR to fix?
(Apologies couldn't reply here earlier)
Hey @yuqian90 , did the proposed solution work for you locally? Do you think we should make a PR to fix?
(Apologies couldn't reply here earlier)
Hi, @kaxil, yeah I can take another look and put up a PR.
@yuqian90 is this issue still relevant?
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
This issue has been closed because it has not received response from the issue author.
When
ExternalTaskMarker
is used together withuser_defined_macros
, we get an unpleasant traceback when trying to clear tasks across DAGs.This example (modified from
example_dags/example_external_task_marker_dag.py
) reproduces the issue. When user click onparent_task
and hit Clear, they get the following traceback because the serialized dag obtained by the webserver has nouser_defined_macros
.Apache Airflow version: Airflow 2.0 or 1.10.* with
store_serialized_dags = True
Kubernetes version (if you are using kubernetes) (use
kubectl version
): NA