search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.21k stars 14.32k forks source link

Build a new main dags page in React #19942

Open bbovenzi opened 2 years ago

bbovenzi commented 2 years ago

We can do a lot more things if the main /dags page was built in React.

Of note, improved sorting/filtering and auto-refresh would be great additions.

To scope: to rethink the whole home page UX from the ground up and see what other big improvements we can bring. In particular I think it would be great to show some basic graphs to give a quick view on what is working/not working.

uranusjr commented 2 years ago

A suggestion (mock-up?) was proposed in #24439:

Home page mockup from #24439

It's pretty useful to highlight the

  • recent dag failures
  • Recently succeeded runs
  • Currently running dags
  • Upcoming dag runs

Optionally to display other metadata like total number of task runs in the last 24 hours etc.

tirkarthi commented 3 months ago

The linked screenshot is from dashboard of prefect probably version 2. Below is prefect 3 screenshot recently released.

I have less experience on UI front but did some work on trying to build an internal dashboard. It's mostly in early stages and presents a live feed like UI with cards and framer motion for transitions. It also presents upcoming dagruns which was also a requested feature internally. It required a new API to fetch 100 task instances per state like running, deferred, etc in the past 24 hours and to give a single compact JSON with only required fields to reduce DB load and also improve rendering. I am yet to optimise the queries and test the scale of it. I guess these changes can be made a plugin to iterate and release faster and also to be available in Airflow 2.10 if needed.

Given that Airflow 3 will have major UI changes I would really love to see a modernized landing page/dashboard and happy to contribute as much as I can.

Branch : https://github.com/tirkarthi/airflow/tree/dashboard-dags

Demo :

https://github.com/user-attachments/assets/f0fd0a45-e494-4405-aa5f-5d7f6f5496e9

Prefect 3 UI as per landing page.

image

bbovenzi commented 3 months ago

Nice work @tirkarthi. I like the upcoming dag runs and seeing them change columns as they run.

I just started the new react UI project. Once it's ready for outside contributions I would love to work together on these ideas for an updated home page. I could imagine it will also involve updating the Cluster Activity page that @pierrejeambrun built too.

We definitely still need a dags list, but that doesn't have to be the home page. How do we merge your dashboard with cluster activity to make an effective home page? Users should be able to customize their dashboards by showing and hiding various panels or using new React Plugins to make their own.