New Graph UI seems to struggle with task-to-task dependencies in nested task groups

[codecae]

Apache Airflow version

2.7.2

What happened

When rendering task-to-task dependencies between tasks that are buried into nested task groups, the UI tends to draw the relationships and node spacing in unusual ways.

What you think should happen instead

The UI should render the graph consistently, regardless of what level you have drilled into the task groups.

How to reproduce

Run this test code to reproduce the issue:

from airflow.decorators import dag, task, task_group
import pendulum
from typing import List, Dict
from airflow.models.baseoperator import BaseOperator

_all_tasks: Dict[str, List[BaseOperator]]={}

@dag(
    schedule=None, 
    start_date=pendulum.datetime(2023,1,1),
)
def test_dag():

    def _generate_task_group_callable(group_id, tasks):
        @task_group(group_id=group_id)
        def _task_group():
            [t() for t in tasks]
        return _task_group

    def _generate_task_callable(task_id, task_set):
        def _generate():
            @task(task_id=task_id)
            def _task():
                pass

            if _all_tasks.get(task_set, None) == None:
                _all_tasks[task_set]=list()

            _all_tasks[task_set].append(_task())
        return _generate

    _set_callables=list()
    for _set in range(0,2):
        _set_id=f"set_{_set}"
        _group_callables_a=list()
        for _nga in range(0,3):
            _nga_id=f"group_{_nga}"
            _group_callables_b=list()
            for _ngb in range(0,5):
                _ngb_id=f"group_{_ngb}"
                _task_callables=list()
                for _task in range(0,10):
                    _task_id=f"task_{_task}"
                    _callable=_generate_task_callable(_task_id, _set_id)
                    _task_callables.append(_callable)
                _group_callable_inner=_generate_task_group_callable(_ngb_id,_task_callables)
                _group_callables_b.append(_group_callable_inner)
            _group_callable_outer=_generate_task_group_callable(_nga_id, _group_callables_b)
            _group_callables_a.append(_group_callable_outer)
        _set_callable=_generate_task_group_callable(_set_id, _group_callables_a)
        _set_callables.append(_set_callable)

    [_s() for _s in _set_callables]

    for i in range(0,len(_all_tasks['set_0'])):
        _all_tasks['set_0'][i] >> _all_tasks['set_1'][i]

test_dag()

Operating System

Debian GNU/Linux 11 (bullseye) -- apache/airflow:slim-2.7.2-python3.10 container

Versions of Apache Airflow Providers

apache-airflow-providers-common-sql==1.7.2 apache-airflow-providers-ftp==3.5.2 apache-airflow-providers-hashicorp==3.5.0 apache-airflow-providers-http==4.5.2 apache-airflow-providers-imap==3.3.2 apache-airflow-providers-sqlite==3.4.3

Deployment

Docker-Compose

Deployment details

No response

Anything else

if the grid is is filtered, it has been observed that the graph renders properly in some scenarios. Doing task-group-to-task-group relationships sometimes renders properly, as well.

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[codecae]

Seems that this may have become buried... Anything I can do to increase awareness? The graph view is generally unusable on larger, more complex dags.

[eladkal]

cc @bbovenzi is these a quick solution to this bug?

[codecae]

While attempting to root out the bug, the only thing I could find that might have any influence would be when the bend points are set explicitly.

https://github.com/apache/airflow/blob/bf9bb72bb631c098bfee34a372aefb332c3ffd64/airflow/www/static/js/dag/details/graph/utils.ts#L209-L211

[codecae]

Due to not being versed enough in react/ts development and debugging, I can't say for sure, but it seemed likely after a cursory review of the logic.