Downstream sensors button

[nivdror]

Description

Hi, We have custom sensors that are based on BaseSensorOperator to link between our dags in our Airlow environment. Right now we have the ability to move backwards via external DAG button when pressing on the sensor in the graph UI.

This is not enough, I would to be able to travel forward as well. I would like the graph to show on which tasks a sensor is listening and tab of connection(with links of course) to those sensors.

We created this feature in our own old airflow but I don't see why this cannot be in apache airflow. Here is an example:

after pressing the downstream sensors button

Use case/motivation

I'm trying to achieve a better way to move/investigate dags in airflow

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[Taragolis]

Are you willing to submit a PR? [ ] Yes I am willing to submit a PR!

We created this feature in our own old airflow but I don't see why this cannot be in apache airflow.

🤔

[Taragolis]

If you would like propose a feature better to provide a PR, otherwise it might take unpredictable time to implement it. In short until someone who want to implement this and might take a time from couple day to decades.

Apache Airflow is an OSS and everyone could propose changes by raising Pull Request, and if it pass code review it would be include into the next minor version of Airflow.

[nivdror]

I'm aware that Airflow is a open source project. However I don't possess any frontend skills nor the time to learn them. So I open a feature request in hope that someone will think the idea is worthy and implement it.

[nathadfield]

@nivdror Maybe you could collaborate with the people who did create this as a custom feature to bring it to the project?

[nivdror]

@nathadfield No, those people have left the company

[nathadfield]

Ok, well, we'll leave this as an open feature request but it might not get any traction without someone else seeing the value and making it happen. Maybe you could try to build some connections with other UI interested community members and collaborating from there?

[nivdror]

@nathadfield thank you. I will see what I can do

[eladkal]

Isn't that exactly what you can see in the dataset view? You can move upstream or downstream

[nivdror]

Hi, we won't use datasets. Our architecture is built mainly on three layers - scheduler layer, context layer and compute layer. Each got it's dependencies and definition. So we see airflow as scheduler which doesn't need to what it's running(spark, MapReduce etc') nor where (CDH, EMR, Databricks etc') nor the data artifacts we output.

This helps us migrate through different providers of Airflow + plus helps us decouple the dependencies. In summary, we don't put all of our eggs in one basket.

[bbovenzi]

We did a bit to show datasets connected upstream and downstream of a DAG on the DAG page. It's not a bad idea to include all types of dependencies like sensors and triggers.