AIP-84 Bootstrap the new Rest API

[pierrejeambrun]

Body

After we decide on our tech stack. Let's get the UI REST API started.

The REST API should also be plugged into the new UI full react app.

MVP PR needs:

• [x] research tech stack, (probably fastapi, what else?) => going with fast api
• [x] Make pydantic v2 and fastapi as core dependencies for airflow 3, remove all v1 / v2 / none code (utils, test, CI). https://github.com/apache/airflow/pull/41857
• [x] migrate one custom endpoint from views.py https://github.com/apache/airflow/pull/41798
  • Let's start with next_run_datasets, its relatively simple and we will still need it
  • We will keep the old UI connected to the old views.py endpoint for now, and delete both later. Then we're not doing extra work just for backward compatibility
• [x] Integrate the new webserver to the airflow CLI (maybe with a warning to discourage usage for now, also in airflow standalone) https://github.com/apache/airflow/pull/41896
• [x] Integrate the new webserver to the Breeze workflow https://github.com/apache/airflow/pull/41896
• [x] Fix Wrong file descriptor with Uvicorn async workers for Gunicorn. Check workers refresh. https://github.com/apache/airflow/pull/41896
• [ ] Add TLS to the API (moved to separate issue, cf note bellow)
• [ ] Handle permissions and access control, POC here https://github.com/apache/airflow/pull/42019 (moved to separate issue, cf note bellow)
• [x] Tests + CI integrations https://github.com/apache/airflow/pull/41957
• [x] develop dev tools for the new API and for the UI to consume the new API (testing, autogeneration) https://github.com/apache/airflow/pull/42098 https://github.com/apache/airflow/pull/42222
• [x] contributor documentation https://github.com/apache/airflow/pull/41903 (brent did it)

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pierrejeambrun]

Done.

HTTPS and Permissions are moved to separate issues. Cf project board.