search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.01k stars 14.27k forks source link

airflow.cfg can't pull secrets from LocalFilesystemBackend #41512

Open thedustinmiller opened 2 months ago

thedustinmiller commented 2 months ago

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When starting an Airflow instance or checking a property via airflow config list/get-value properties that have the _secret suffix return their literal value and the main property isn't set at all. The default values instead are applied.

What you think should happen instead?

According to the docs, the following snippet should retrieve the key sql_alchemy_conn from the secret backend (the local filesystem one, in my case) and assign it to the sql_alchemy_conn under the [database] section. 

[database]
sql_alchemy_conn_secret = sql_alchemy_conn

https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html

How to reproduce

Reproducing requires only configuring the following properties in airflow.cfg and the .env file in the same directory. The secret backend still works from within a DAG, however, it just seems to not be invoked while parsing the airflow.cfg. I tried bringing the [secrets] section to the top of the airflow.cfg, since by default it comes after [database], but that had no effect.

[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "./.env"}

[database]
sql_alchemy_conn_secret = sql_alchemy_conn

and an .env file (or yaml or json) like this

sql_alchemy_conn=postgresql://airflow_user@localhost/airflow_db

Operating System

Arch linux 6.10.4-arch2-1

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.7.2 apache-airflow-providers-common-io==1.3.2 apache-airflow-providers-common-sql==1.14.2 apache-airflow-providers-fab==1.2.2 apache-airflow-providers-ftp==3.10.0 apache-airflow-providers-http==4.12.0 apache-airflow-providers-imap==3.6.1 apache-airflow-providers-smtp==1.7.1 apache-airflow-providers-sqlite==3.8.1

Deployment

Virtualenv installation

Deployment details

venv setup from python 3.12.4 pip install "apache-airflow[celery]==2.9.3" --constraint "./constraints-3.8.txt"

Anything else?

This is from a completely fresh installation. The issue also appears on an ARM Mac. I don't have the other possible secret backends set up, so I don't know how those behave. I can also understand that this might be a documentation issue and this isn't supported anymore, since the secrets backend would be retrieving secrets for the same file it is configured in itself.

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 2 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.