search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.75k stars 14.22k forks source link

Error rendering template: '_GeneratorContextManager' object has no attribute 'query' #39162

Open raphaelauv opened 6 months ago

raphaelauv commented 6 months ago

Apache Airflow version

2.9.0

What happened?

on a KPO mapped task , the rendered template fail

Screenshot from 2024-04-22 09-46-17

[2024-04-22T07:38:22.812+0000] {app.py:1744} ERROR - Exception on /extra_links [GET]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", line 252, in decorated
    return _has_access(
           ^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", line 163, in _has_access
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 79, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py", line 3111, in extra_links
    url = task.get_extra_links(ti, link_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/abstractoperator.py", line 493, in get_extra_links
    return link.get_link(self.unmap(None), ti_key=ti.key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/trigger_dagrun.py", line 69, in get_link
    return build_airflow_url_with_query(query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/helpers.py", line 253, in build_airflow_url_with_query
    return flask.url_for(f"Airflow.{view}", **query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/helpers.py", line 256, in url_for
    return current_app.url_for(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2034, in url_for
    return self.handle_url_build_error(error, endpoint, values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2023, in url_for
    rv = url_adapter.build(  # type: ignore[union-attr]
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/routing/map.py", line 917, in build
    raise BuildError(endpoint, values, method, self)
werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'Airflow.grid' with values ['base_date']. Did you forget to specify values ['dag_id']?
[2024-04-22T07:39:41.734+0000] {base.py:84} INFO - Using connection ID 'XXXXXXX' for task execution.
[2024-04-22T07:39:41.735+0000] {connection_wrapper.py:384} INFO - AWS Connection (conn_id='XXXXXXX', conn_type='aws') credentials retrieved from login and password.
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_start_date` method.

/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_ti` method.

[2024-04-22T07:44:28.101+0000] {base.py:84} INFO - Using connection ID 'XXXXXXX' for task execution.
[2024-04-22T07:44:28.103+0000] {connection_wrapper.py:384} INFO - AWS Connection (conn_id='XXXXXXX', conn_type='aws') credentials retrieved from login and password.
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_start_date` method.

/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_ti` method.

/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_start_date` method.

/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_ti` method.

[2024-04-22T07:48:04.801+0000] {views.py:1368} INFO - Retrieving rendered templates.
[2024-04-22T07:48:04.875+0000] {task_instance_session.py:38} WARNING - No task session set for this task. Continuing but this likely causes a resource leak.
[2024-04-22T07:48:04.875+0000] {task_instance_session.py:39} WARNING - Please report this and stacktrace below to https://github.com/apache/airflow/issues
[2024-04-22T07:48:04.880+0000] {task_instance_session.py:41} WARNING - File: "<frozen runpy>", 198 , in _run_module_as_main
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:41} WARNING - File: "<frozen runpy>", 88 , in _run_code
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/__main__.py", 7 , in <module>
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:43} WARNING -   run()
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", 67 , in run
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:43} WARNING -   WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/app/base.py", 236 , in run
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:43} WARNING -   super().run()
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/app/base.py", 72 , in run
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:43} WARNING -   Arbiter(self).run()
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 202 , in run
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:43} WARNING -   self.manage_workers()
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 571 , in manage_workers
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:43} WARNING -   self.spawn_workers()
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 642 , in spawn_workers
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:43} WARNING -   self.spawn_worker()
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 609 , in spawn_worker
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:43} WARNING -   worker.init_process()
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/base.py", 142 , in init_process
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:43} WARNING -   self.run()
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 126 , in run
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:43} WARNING -   self.run_for_one(timeout)
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 70 , in run_for_one
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:43} WARNING -   self.accept(listener)
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 32 , in accept
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:43} WARNING -   self.handle(listener, client, addr)
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 135 , in handle
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:43} WARNING -   self.handle_request(listener, req, client, addr)
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 178 , in handle_request
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:43} WARNING -   respiter = self.wsgi(environ, resp.start_response)
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 2552 , in __call__
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:43} WARNING -   return self.wsgi_app(environ, start_response)
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/middleware/proxy_fix.py", 187 , in __call__
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:43} WARNING -   return self.app(environ, start_response)
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 2529 , in wsgi_app
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:43} WARNING -   response = self.full_dispatch_request()
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 1823 , in full_dispatch_request
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:43} WARNING -   rv = self.dispatch_request()
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 1799 , in dispatch_request
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:43} WARNING -   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", 252 , in decorated
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:43} WARNING -   return _has_access(
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", 163 , in _has_access
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:43} WARNING -   return func(*args, **kwargs)
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", 79 , in wrapper
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:43} WARNING -   return func(*args, session=session, **kwargs)
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py", 1410 , in rendered_templates
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:43} WARNING -   ti.get_rendered_template_fields(session=session)
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", 76 , in wrapper
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:43} WARNING -   return func(*args, **kwargs)
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", 3041 , in get_rendered_template_fields
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:43} WARNING -   self.render_templates()
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", 3079 , in render_templates
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:43} WARNING -   original_task.render_template_fields(context, jinja_env)
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/mappedoperator.py", 827 , in render_template_fields
[2024-04-22T07:48:04.892+0000] {task_instance_session.py:43} WARNING -   session = get_current_task_instance_session()
[2024-04-22T07:48:04.892+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/task_instance_session.py", 40 , in get_current_task_instance_session
[2024-04-22T07:48:04.892+0000] {task_instance_session.py:43} WARNING -   for filename, line_number, name, line in traceback.extract_stack():

Are you willing to submit PR?

Code of Conduct

RNHTTR commented 6 months ago

Can you provide reproduction steps? I'm running a mapped KPO on Airflow 2.9.0 and the rendered template view is fine

image
raphaelauv commented 6 months ago

I do not know how to reproduce , only context I have the task was run in 2.8.0-pyt11 and stack is currently in 2.9.0-py12

github-actions[bot] commented 5 months ago

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

mro-dmoura commented 5 months ago

I've been experiencing this error in mapped tasks. Many mapped tasks are failing with "[2024-05-08, 03:07:51 CDT] {scheduler_job_runner.py:769} ERROR - Executor reports task instance <TaskInstance: scheduled__2024-05-07T08:00:00+00:00 [queued]> finished (failed) although the task says it's queued. (Info: None) Was the task killed externally?". When investigating, the rendered template is showing the error: "Error rendering template: '_GeneratorContextManager' object has no attribute 'query'". This started happening after upgrading from 2.8.1 to 2.9.1. Any ideas?

RNHTTR commented 5 months ago

@mro-dmoura A couple questions:

  1. Is this happening consistently or intermittently?
  2. Is this a mapped KubernetesPodOperator?
raphaelauv commented 5 months ago

It's only sometimes.

the same task ( mapped ) is not always problematic for all dag_run

sukalpv commented 1 month ago

facing this since upgrading to 2.9.1. The code was working fine with airflow version 2.6.3. Using PythonVirtualEnvOperator