[bitnami/airflow] latest container - env setup scripts failing with sed error when AIRFLOW_LDAP_ENABLE is enabled

ITJamie commented 1 year ago

Name and Version

bitnami/airflow:2-debian-11 - 717b69e002bc

What architecture are you using?

None

What steps will reproduce the bug?

attempt to use the bitnami airflow container with AIRFLOW_LDAP_ENABLE=yes

What is the expected behavior?

previous version (2.6.3):

airflow_1           | airflow 12:09:36.89 INFO  ==> ** Starting Airflow setup **
airflow_1           | airflow 12:09:36.90 INFO  ==> Validating settings in POSTGRESQL_CLIENT_* env vars
airflow_1           | airflow 12:09:36.95 INFO  ==> Initializing Airflow ...
airflow_1           | airflow 12:09:36.96 INFO  ==> No injected configuration file found. Creating default config file
airflow_1           | airflow 12:09:38.25 INFO  ==> Configuring Airflow webserver authentication
airflow_1           | airflow 12:09:38.26 INFO  ==> Enabling LDAP authentication
airflow_1           | airflow 12:09:38.35 INFO  ==> Configuring Airflow database
airflow_1           | airflow 12:09:38.41 INFO  ==> Configuring Celery Executor
airflow_1           | airflow 12:09:38.46 INFO  ==> Trying to connect to the database server
airflow_1           | airflow 12:09:43.13 INFO  ==> Upgrading database schema
airflow_1           | airflow 12:09:54.99 INFO  ==> ** Airflow setup finished! **
airflow_1           |
airflow_1           | airflow 12:09:55.02 INFO  ==> ** Starting Airflow **

What do you see instead?

airflow_1           | airflow 11:51:54.89 INFO  ==> ** Starting Airflow setup **
airflow_1           | airflow 11:51:54.90 INFO  ==> Validating settings in POSTGRESQL_CLIENT_* env vars
airflow_1           | airflow 11:51:54.95 INFO  ==> Initializing Airflow ...
airflow_1           | airflow 11:51:54.96 INFO  ==> No injected configuration file found. Creating default config file
airflow_1           | airflow 11:51:56.12 INFO  ==> Configuring Airflow webserver authentication
airflow_1           | airflow 11:51:56.13 INFO  ==> Enabling LDAP authentication
airflow_1           | sed: can't read /opt/bitnami/airflow/webserver_config.py: No such file or directory
airflow_1           | airflow 11:51:56.94
airflow_1           | airflow 11:51:56.94 Welcome to the Bitnami airflow container
airflow_airflow-worker_1 exited with code 1
airflow_1           | airflow 11:51:56.94 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow_1           | airflow 11:51:56.95 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow_1           | airflow 11:51:56.95
airflow_1           | airflow 11:51:56.96 INFO  ==> Enabling non-root system user with nss_wrapper

after the sed error the setup script stops and skips other steps after the fact (note it doesn't reach ** Airflow setup finished! ** stage and instead skips to starting airflow with a default configuration file

setting AIRFLOW_LDAP_ENABLE to no lets the setup script complete.

ITJamie commented 1 year ago

this worked fine on 2.6.3

but broke as of 2.7.0 https://hub.docker.com/layers/bitnami/airflow/2.7.0/images/sha256-8b87c796104cf8fca1a8469c63f8fd03f61934cc9172cb4f460b1d438015213f?context=explore

fevisera commented 12 months ago

Hi @ITJamie,

Are you using a specific values.yaml or docker-compose.yaml configuration when attempting to run airflow container?

I attempted to replicate the issue, but I encountered a different error message:

$ docker run --rm -e AIRFLOW_LDAP_ENABLE=yes --pull=always bitnami/airflow
latest: Pulling from bitnami/airflow
Digest: sha256:45a00285b2af2e97ed2505f86f37844b24dc417272bde99aefc98d4151ed9c86
Status: Image is up to date for bitnami/airflow:latest
airflow 08:00:34.08
airflow 08:00:34.09 Welcome to the Bitnami airflow container
airflow 08:00:34.09 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow 08:00:34.09 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow 08:00:34.09
airflow 08:00:34.10 INFO  ==> Enabling non-root system user with nss_wrapper
airflow 08:00:34.12 INFO  ==> ** Starting Airflow setup **
airflow 08:00:34.13 INFO  ==> Validating settings in POSTGRESQL_CLIENT_* env vars
airflow 08:00:34.18 ERROR ==> Missing AIRFLOW_LDAP_URI
airflow 08:00:34.18 ERROR ==> Missing AIRFLOW_LDAP_SEARCH
airflow 08:00:34.18 ERROR ==> Missing AIRFLOW_LDAP_UID_FIELD
airflow 08:00:34.18 ERROR ==> Missing AIRFLOW_LDAP_BIND_USER
airflow 08:00:34.19 ERROR ==> Missing AIRFLOW_LDAP_BIND_PASSWORD
airflow 08:00:34.19 ERROR ==> Missing AIRFLOW_LDAP_ROLES_MAPPING
airflow 08:00:34.20 ERROR ==> Missing AIRFLOW_LDAP_USER_REGISTRATION_ROLE

Is it possible that one of these additional environment variables is being missed?

ITJamie commented 12 months ago

Theres an example set of values in another report of this. It is not due to missing values

https://github.com/bitnami/charts/issues/19000

fevisera commented 12 months ago

Hi @ITJamie,

I have relocated this issue to "containers" as you mentioned in https://github.com/bitnami/charts/issues/19000 it has been produced using a docker-compose.yml file.

I was able to reproduce the error by using docker-compose-ldap.yml, but I had to include some additional modifications (https://github.com/bitnami/containers/pull/48236).

$ docker-compose -f docker-compose-ldap.yml logs -f airflow
...
airflow-airflow-1  | airflow 08:06:25.94 INFO  ==> ** Starting Airflow setup **
airflow-airflow-1  | airflow 08:06:25.95 INFO  ==> Validating settings in POSTGRESQL_CLIENT_* env vars
airflow-airflow-1  | airflow 08:06:26.02 INFO  ==> Initializing Airflow ...
airflow-airflow-1  | airflow 08:06:26.04 INFO  ==> No injected configuration file found. Creating default config file
airflow-airflow-1  | airflow 08:06:27.22 INFO  ==> Configuring Airflow webserver authentication
airflow-airflow-1  | airflow 08:06:27.23 INFO  ==> Enabling LDAP authentication
airflow-airflow-1  | sed: can't read /opt/bitnami/airflow/webserver_config.py: No such file or directory
$
$ docker-compose -f docker-compose-ldap.yml ps
NAME                          COMMAND                  SERVICE             STATUS              PORTS
airflow-airflow-1             "/opt/bitnami/script…"   airflow             exited (2)          
airflow-airflow-scheduler-1   "/opt/bitnami/script…"   airflow-scheduler   exited (1)          
airflow-airflow-worker-1      "/opt/bitnami/script…"   airflow-worker      exited (1)          
airflow-openldap-1            "/opt/bitnami/script…"   openldap            running             0.0.0.0:1389->1389/tcp, 1636/tcp
airflow-postgresql-1          "/opt/bitnami/script…"   postgresql          running             5432/tcp
airflow-redis-1               "/opt/bitnami/script…"   redis               running             6379/tcp

I will create an internal task in order to review this and move the issue to hold until we have updates.

Thank you for reporting this!

bitnami / containers