search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.17k stars 14.32k forks source link

Create DAGs Dynamically using REST API #21266

Closed mashirali closed 2 years ago

mashirali commented 2 years ago

Description

I want to add an API endpoint which creates single or multiple dags dynamically using the API.

Use case/motivation

It can work as a backend for various ETL pipelines.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

potiuk commented 2 years ago

This is impossible now. DAGs must be created as Python files in the DAGs folder.

There is a discussion of it (not very active) and possibly in the future it will be implemented - one of the incarnations of it are here https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher - but this is a fundamental change in how Airflow would work and we are not going to implement it any time soon (and if we do - this will be a lot of discussions and AIP). If you would like to raise discussion on it and get interest - feel free to start discussion in the devlist - because that's the only place where such fundamental changes might be raised.

You find the links to the devlist here: https://airflow.apache.org/community/

For now I am closing it becaue this won't happen via "feature request". This is much more than a feature. This is changing how Airflow works entirely.