apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.48k stars 14.37k forks source link

New Graph UI seems to struggle with task-to-task dependencies in nested task groups #35301

Open codecae opened 1 year ago

codecae commented 1 year ago

Apache Airflow version

2.7.2

What happened

When rendering task-to-task dependencies between tasks that are buried into nested task groups, the UI tends to draw the relationships and node spacing in unusual ways.

image image image

What you think should happen instead

The UI should render the graph consistently, regardless of what level you have drilled into the task groups.

How to reproduce

Run this test code to reproduce the issue:

from airflow.decorators import dag, task, task_group
import pendulum
from typing import List, Dict
from airflow.models.baseoperator import BaseOperator

_all_tasks: Dict[str, List[BaseOperator]]={}

@dag(
    schedule=None, 
    start_date=pendulum.datetime(2023,1,1),
)
def test_dag():

    def _generate_task_group_callable(group_id, tasks):
        @task_group(group_id=group_id)
        def _task_group():
            [t() for t in tasks]
        return _task_group

    def _generate_task_callable(task_id, task_set):
        def _generate():
            @task(task_id=task_id)
            def _task():
                pass

            if _all_tasks.get(task_set, None) == None:
                _all_tasks[task_set]=list()

            _all_tasks[task_set].append(_task())
        return _generate

    _set_callables=list()
    for _set in range(0,2):
        _set_id=f"set_{_set}"
        _group_callables_a=list()
        for _nga in range(0,3):
            _nga_id=f"group_{_nga}"
            _group_callables_b=list()
            for _ngb in range(0,5):
                _ngb_id=f"group_{_ngb}"
                _task_callables=list()
                for _task in range(0,10):
                    _task_id=f"task_{_task}"
                    _callable=_generate_task_callable(_task_id, _set_id)
                    _task_callables.append(_callable)
                _group_callable_inner=_generate_task_group_callable(_ngb_id,_task_callables)
                _group_callables_b.append(_group_callable_inner)
            _group_callable_outer=_generate_task_group_callable(_nga_id, _group_callables_b)
            _group_callables_a.append(_group_callable_outer)
        _set_callable=_generate_task_group_callable(_set_id, _group_callables_a)
        _set_callables.append(_set_callable)

    [_s() for _s in _set_callables]

    for i in range(0,len(_all_tasks['set_0'])):
        _all_tasks['set_0'][i] >> _all_tasks['set_1'][i]

test_dag()

Operating System

Debian GNU/Linux 11 (bullseye) -- apache/airflow:slim-2.7.2-python3.10 container

Versions of Apache Airflow Providers

apache-airflow-providers-common-sql==1.7.2 apache-airflow-providers-ftp==3.5.2 apache-airflow-providers-hashicorp==3.5.0 apache-airflow-providers-http==4.5.2 apache-airflow-providers-imap==3.3.2 apache-airflow-providers-sqlite==3.4.3

Deployment

Docker-Compose

Deployment details

No response

Anything else

if the grid is is filtered, it has been observed that the graph renders properly in some scenarios. Doing task-group-to-task-group relationships sometimes renders properly, as well.

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

codecae commented 9 months ago

Seems that this may have become buried... Anything I can do to increase awareness? The graph view is generally unusable on larger, more complex dags.

eladkal commented 8 months ago

cc @bbovenzi is these a quick solution to this bug?

codecae commented 8 months ago

While attempting to root out the bug, the only thing I could find that might have any influence would be when the bend points are set explicitly.

https://github.com/apache/airflow/blob/bf9bb72bb631c098bfee34a372aefb332c3ffd64/airflow/www/static/js/dag/details/graph/utils.ts#L209-L211

codecae commented 8 months ago

Due to not being versed enough in react/ts development and debugging, I can't say for sure, but it seemed likely after a cursory review of the logic.