Showing DAGs in folder like structure

[msumit]

Description

The DAGs main page looks messy when we've too many dags. What if we can create a soft foldering around it? I mean a UI-only folder structure to group DAGs. There could be 2 possible ways to group (or maybe more). The first way is grouping via subpaths under DAGS_FOLDER where the actual dag file is present. The second way is via tags. There could be a toggle button at the top to get the flat list structure as we've at present.

@bradarndt @ryanahamilton what do you guys think?

Use case/motivation

This feature would help keep the DAGs main page neat and clean. It would also motivate the users to arrange the dags as per their functionalities.

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[potiuk]

+1

[potiuk]

Also this discussion might be relevant https://github.com/apache/airflow/discussions/18551 - where users expressed an interest in seeing "related" files. This seems quite connected (especially if we can assume that subpaths can be used for groupping and showing relevant files).

This sounds to me like a small AIP waiting to be written

[msumit]

18551 is an interesting ask, but after DAG serialization I think it's quite difficult to achieve.

[potiuk]

18551 is an interesting ask, but after DAG serialization I think it's quite difficult to achieve.

True. unless we serialize more :)

[erdos2n]

@potiuk we could serialize the relative path of the dag_folder? Thoughts?

[potiuk]

This is more complex. WebServer does not need (and should not need) access to the dag folder, so you'd have to serialize the whole folder - which might containe also other DAG files for example and a bit too much to serialize.

After thinking a bit more, I think the only "sane" way to add extra files is to specify what to serialize together with the dag - like a glob of the files to add to serialized form. In which case it will be very diffeent from what @msumit idea is with the folder structre - which will be quite different task (and should be discussed in #18551).

[msumit]

I believe implementing 18551 shouldn't be Airflow's responsibility. This is something that can be supported by Airflow SAAS vendors or even Git-based solutions. For example, years back we built a DAG Explorer to achieve something very similar.

[potiuk]

I believe implementing 18551 shouldn't be Airflow's responsibility. This is something that can be supported by Airflow SAAS vendors or even Git-based solutions. For example, years back we built a DAG Explorer to achieve something very similar.

I think you are right @msumit . I have no very strong opinion there. But you could theoretically even write a custom plugin to do that, to pull your "linked" sources" from somewhere else - not necessarily from Airflow DB - but straight from the source of the data. Serializing those external "data" to the database just to show it in the UI seems like huge waste.

[msumit]

@potiuk totally agree with you.

[draev]

Who is in charge ? Should we wait somewhen for upgrading UI and be able to group DAGs in folder structure ? Have more than 500 DAGs and it became more difficult to manage it.

[potiuk]

Whoever picks up the task. It can be you for example. This is an Open-Source project with > 2.500 contributors - anyone can pick up such task and implement it.

[draev]

Fully agree with you, but taking in consideration the duration of this open ticket, i thought that somebody possible started to do something.

As well, I see that this issue is not so popular.. maybe there is another way to organize your DAGs (except tags), I just missed out 🤔

[ammarchalifah-bolt]

I'm very interested with this feature. Any update with this?

[potiuk]

I'm very interested with this feature. Any update with this?

See the comments above.

Generally - if there are no comments or information that somoene works on it - likely no-one works on it, otherwise there would be comments or information (stating the bobvioous - this is open-source project run under solid - transparent governance - so nothing here is "hidden").

But if you are very interested - it's up for grabs. You might contribute it, or maybe your company can pay someone to contribute it if you do not have enough skills and time.

In open source like this, fetures get implemented when someone, well, implements them. And people implement features if they get interest in it, or are paid for contributing or simply they feel like doing it. Airlfow is an open-source project that anyone can contribute to, so there is no single place or roadmap where you can check "if something will happen".

The most "sure" way of making sure that things get implemented is taking the lead and welll implementing it (with your or other hands you engage).

So if you are interested in something - by all means - do, contribute.