search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.21k stars 13.76k forks source link

Add declarative configuration to define custom roles #38347

Open ketozhang opened 3 months ago

ketozhang commented 3 months ago

Description

Add declarative configuration to define custom roles in either airflow.cfg or webserver_config.py.

Use case/motivation

Currently the only way to define custom roles is through the CLI or UI. This isn't ideal for those deploying configuration through a repository (e.g., GitOps).

It seems like this feature is already possible with webserver_config.py, however not documented here. If so, I will change the request... https://airflow.apache.org/docs/apache-airflow/stable/security/webserver.html

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

potiuk commented 3 months ago

Sure. If you have a good idea how to do it in a robust way - proposing a solution, especially one that will not overwrite existing DB and will allow to both declaratively upload roles from a source and manually modify them, and keep those local modificiattions - PRs are most welcome.

Just make sure when you submit the PR to propose a solution for this conundrum where you have both manually modifiable data and uploadable from the externally depoyable sources in declarative way. So far all the efforts failed on that step, but maybe you will be able to propose something.

BTW. once you find a solution, also similar thing should be done from connections, users and variables - those suffer from the same issue and if you find and get approval to solve it for roles, applying similar approach for the others is a natural next step.

ketozhang commented 3 months ago

Thanks and yep, it's a classic problem most devops folks have to deal with. It's heavily dependent if you want to lock down configs post-deploy and if DB should be overwritten during a redeploy.

What's currently being done for connections and variables to solve that issue? They both already have declarative equivalents.

So far all the efforts failed on that step

How have they failed?

potiuk commented 3 months ago

How have they failed?

To solve all the edge cases and the fact that users can modify the DB after it has been created and how to consistently mantain source of the truth.t But maybe your proposal will fix it. Looking forward to it.

potiuk commented 3 months ago

What's currently being done for connections and variables to solve that issue? They both already have declarative equivalents.

No - they don't - same problem. When you allow to edit data over the UI and want to also upload it at the deployment time, you need to solve the problem of data being modified both manually and through declarative sources at deplpyment time