search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.31k forks source link

AIP-84 Migrate public endpoints from api_connexion to the new rest api #42370

Open pierrejeambrun opened 1 month ago

pierrejeambrun commented 1 month ago

Body

https://github.com/apache/airflow/issues/42366 is taking care of migrating private endpoints.

This issue focus on migrating all the public ones. Ideally they shouldn't change too much and be feature rich compare to airflow 2.x.

An exemple PR can be found here https://github.com/apache/airflow/pull/42196

Tasks (total ~ 81 endpoints)

Backfill (6 endpoints) @dstandish

Config (2 endpoints) @rawwar

Connection @bugraoz93 (6 endpoints)

DAG (5 endpoints)

Dag Parsing (1 endpoint) @prabhusneha

Dag Runs @rawwar (9 endpoints)

Dag Source (1 endpoint)

Dag Stats (1 endpoint)

Dag Warnings (1 endpoint) @rawwar

Dataset / Assets (10 endpoints) @amoghrajesh

Event Logs (2 endpoints)

Extra Link (1 endpoint) @prabhusneha

Monitor (2 endpoints)

Import Error (2 endpoints) @jason810496

Log (1 endpoint) @prabhusneha

Plugin (1 endpoint)

Pool (5 endpoints) @pierrejeambrun

Provider (1 endpoint)

Task (2 endpoints) @omkar-foss

Task Instance (15 endpoints) @pierrejeambrun

Variables (5 endpoints)

XCom (2 endpoints)

Committer

omkar-foss commented 1 month ago

Trimming down views.py is very tempting 😄

May I pick this up? If yes, please assign this issue to me. Thanks!

pierrejeambrun commented 1 month ago

Hello @omkar-foss,

views.py are mostly 'rendered' front end that will get replaced by the new UI react, in views.py you also find private/UI REST route, most of them are with the /object/ prefix, such as def grid_data. If you want to work on those UI dedicated endpoints you can that would be part of https://github.com/apache/airflow/issues/42366.

This issue focuses on the public endpoints, those are located under the api_connexion folder.

In anycase, just mention the endpoint you are working on, so people do not do the same ones, and I will update the description to track that :). I think we should do one at the time.

omkar-foss commented 1 month ago

Hi @pierrejeambrun, thank you for explaining it to me, much appreciate it.

I'll start migrating the public endpoints, beginning with migration of the DAG Details API (get_dag_details) in api_connexion/endpoints/dag_endpoint.py.

Since this is a meta issue (missed that previously!), I'll create the issue/PR corresponding to the endpoint(s) that I'll be working on and link to this issue (like usual).

P.S: views.py will have to wait :)

bugraoz93 commented 1 month ago

Thanks for creating the ticket @pierrejeambrun! Let me take connection endpoints

pierrejeambrun commented 1 month ago

Great, thanks @bugraoz93

bugraoz93 commented 1 month ago

Thanks for assigning! Let me save you from regular updates :) I created issues for all the endpoints in connections.

pierrejeambrun commented 1 month ago

Great, thanks @bugraoz93. I just updated the meta issue with those new additions. Let me know if you plan to work on all of them so I can assign you, or just some of them so I can label the others with good first issue and let other contributors grab them :)

bugraoz93 commented 1 month ago

I am planning to work on all of them @pierrejeambrun. if you can assign them to me, that would be great! Thanks :)

omkar-foss commented 1 month ago

Hey folks! After https://github.com/apache/airflow/pull/42631 (open for review), I can pick up these two next in order:

  1. get_dag - https://github.com/apache/airflow/issues/42652
  2. delete_dag - https://github.com/apache/airflow/issues/42650

With these 2 done, I suppose we should have the dags.py migration complete. When you find some time, please update this meta and assign these to me, thank you! 😃

bbovenzi commented 1 month ago

@omkar-foss Assigned. Thanks for picking these issues up!

rawwar commented 1 month ago

I am working on the following

  1. get_connection - #42674
  2. get_connections - #42677
omkar-foss commented 1 month ago

Hey folks, just a heads-up on Pydantic aliases - try not to use AliasGenerator as it breaks Airflow lowest direct dependency resolution tests with a TypeError because lower versions of Pydantic wouldn't be having this patch. The TypeError looks like alias_generator ... must return str, not ... (for more info see this issue).

Can use either AliasChoices or AliasPath as those work fine. I faced this issue in https://github.com/apache/airflow/pull/42631 and spent a while on it, so just thought should let you all know!

potiuk commented 1 month ago

Hey folks, just a heads-up on Pydantic aliases - try not to use AliasGenerator as it breaks Airflow lowest direct

We can also update the minimum version of Pydantic to the version that is fixed. This is the feature of the "lowest direct" tests that they detect such minimum requirements are not met. So maybe you can figure out which is the minimum version and we set it @omkar-foss ?

omkar-foss commented 1 month ago

This is the feature of the "lowest direct" tests that they detect such minimum requirements are not met.

It's a very cool feature! Helping to find issues we normally wouldn't find during regular unit or integration tests.

We can also update the minimum version of Pydantic to the version that is fixed. So maybe you can figure out which is the minimum version and we set it

That would be great, the minimum Pydantic version that is fixed is v2.6.4, this commit for quick reference.

potiuk commented 1 month ago

That would be great, the minimum Pydantic version that is fixed is v2.6.4, this commit for quick reference.

Added min version for Pydantic in https://github.com/apache/airflow/pull/42694

pierrejeambrun commented 1 month ago

@rawwar

I am working on the following

get_connection - https://github.com/apache/airflow/pull/42674 get_connections - AIP-84 Migrate the public endpoint Get Connections to FastAPI #42677`

Thanks for your contribution and help on AIP-84, as mentioned above, I think @bugraoz93 is already working on connections endpoints, or is planning to do so. Feel free to choose any other endpoints that looks cool to you, under api_connexions/endpoints. Besides dag_endpoint and connection_endpoint eveything is up for a grab, just let me know and I will assign you :)

pierrejeambrun commented 1 month ago

Assigning myself to variables

omkar-foss commented 1 month ago

I've created new issues for sections Dag Source (1 endpoint), Dag Stats (1 endpoint), Task (2 endpoints) and Version (1 endpoint). List as below:

  1. https://github.com/apache/airflow/issues/42876
  2. https://github.com/apache/airflow/issues/42877
  3. https://github.com/apache/airflow/issues/42874
  4. https://github.com/apache/airflow/issues/42875
  5. https://github.com/apache/airflow/issues/42879

Please assign these to me, I'll pick these up one by one in the next few days, right after https://github.com/apache/airflow/issues/42650. Cheers.

pierrejeambrun commented 1 month ago

@omkar-foss assigned and meta issue updated, thanks :)

michaeljs-c commented 1 month ago

Hi @pierrejeambrun I've added issues for XCom, please assign to me 🙂 thanks

pierrejeambrun commented 1 month ago

Hi @pierrejeambrun I've added issues for XCom, please assign to me 🙂 thanks

Done, and meta task is updated.

Thanks for taking those ones @michaeljs-c!

jason810496 commented 3 weeks ago

Hi @pierrejeambrun, I can take on the migrations for Event Logs:

  1. Get Event Log
  2. Get Event Logs

However, none of the current issue form templates fit a task issue (the Task and Meta template tags the issue with kind:meta instead of kind:feature). Could you please create the issue and assign it to me? Thanks!

jason810496 commented 2 weeks ago

Hi @pierrejeambrun, I can handle the migrations for Import Error

  1. get_import_error
  2. get_import_errors

I’m still encountering the issue where I can’t create the correct issue type. Could you please create it and assign it to me? Thanks!

rawwar commented 2 weeks ago

I've unassigned myself for the config endpoint issue. If no one takes it, I'll pick it up once I finish DagRun and DagWarning endpoints. Issue: https://github.com/apache/airflow/issues/42745 <- If anyone wants to take this, please let me know and I can assign it to you.

kandharvishnuu commented 1 week ago

@pierrejeambrun I am working on get taskinstance try endpoint

omkar-foss commented 1 week ago

Hi @pierrejeambrun, could you please assign these TI-related endpoints to me in above issue description:

GET:

POST:

PUT:

pierrejeambrun commented 1 week ago

Hey omkar, as you can see in I'm already assigned to the TaskInstances endpoint.

Thanks for creating the issue, we can split the work on that matter, i'll assign some of them to you maybe all the PUT/POST as I am already working on GET.

pierrejeambrun commented 1 week ago

You're assigned, issue updated, let me know if that shounds good for you :).

omkar-foss commented 1 week ago

You're assigned, issue updated, let me know if that shounds good for you :).

Yes sure, that works. Thank you :)