search

bitnami / containers

Bitnami container images
https://bitnami.com
Other
3.28k stars 4.76k forks source link

airflow-scheduler and airflow-worker can not connct to webserver #50336

Closed kwang1971 closed 11 months ago

kwang1971 commented 11 months ago

Name and Version

bitnami/airflow:2

What architecture are you using?

amd64

What steps will reproduce the bug?

use the following docker-compose.yml to perssiting data to host directory.

# Copyright VMware, Inc.
# SPDX-License-Identifier: APACHE-2.0

version: '2'
services:
  postgresql:
    image: 'bitnami/postgresql:15'
    environment:
      - POSTGRESQL_DATABASE=bitnami_airflow
      - POSTGRESQL_USERNAME=bn_airflow
      - POSTGRESQL_PASSWORD=bitnami1
    volumes:
      - D:/projects/airflow/bitnamiImage/PersistingSolution/postgresql-persistence:/bitnami/postgresql
  redis:
    image: 'bitnami/redis:7.0'
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - D:/projects/airflow/bitnamiImage/PersistingSolution/redis-persistence:/bitnami
  airflow-worker:
    image: bitnami/airflow-worker:2
    environment:
      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
      - AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
      - AIRFLOW_EXECUTOR=CeleryExecutor
      - AIRFLOW_DATABASE_NAME=bitnami_airflow
      - AIRFLOW_DATABASE_USERNAME=bn_airflow
      - AIRFLOW_DATABASE_PASSWORD=bitnami1
      - AIRFLOW_LOAD_EXAMPLES=yes
  airflow-scheduler:
    image: bitnami/airflow-scheduler:2
    environment:
      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
      - AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
      - AIRFLOW_EXECUTOR=CeleryExecutor
      - AIRFLOW_DATABASE_NAME=bitnami_airflow
      - AIRFLOW_DATABASE_USERNAME=bn_airflow
      - AIRFLOW_DATABASE_PASSWORD=bitnami1
      - AIRFLOW_LOAD_EXAMPLES=yes
  airflow:
    image: bitnami/airflow:2
    environment:
      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
      - AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
      - AIRFLOW_EXECUTOR=CeleryExecutor
      - AIRFLOW_DATABASE_NAME=bitnami_airflow
      - AIRFLOW_DATABASE_USERNAME=bn_airflow
      - AIRFLOW_DATABASE_PASSWORD=bitnami1
      - AIRFLOW_PASSWORD=bitnami123
      - AIRFLOW_USERNAME=user
      - AIRFLOW_EMAIL=user@example.com
    ports:
      - '8080:8080'

What is the expected behavior?

system run successfully

What do you see instead?

schedule

airflow-scheduler 23:40:53.28
airflow-scheduler 23:40:53.29 Welcome to the Bitnami airflow-scheduler container
airflow-scheduler 23:40:53.29 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-scheduler 23:40:53.29 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-scheduler 23:40:53.29
airflow-scheduler 23:40:53.30 INFO  ==> Enabling non-root system user with nss_wrapper
airflow-scheduler 23:40:53.31 INFO  ==> ** Starting Airflow setup **
airflow-scheduler 23:40:53.33 INFO  ==> No injected configuration file found. Creating default config file
airflow-scheduler 23:40:54.33 INFO  ==> Configuring Airflow database
airflow-scheduler 23:40:54.35 INFO  ==> Configuring Celery Executor
airflow-scheduler 23:40:54.36 INFO  ==> Trying to connect to the database server
airflow-scheduler 23:40:55.55 INFO  ==> Waiting for Airflow Webserver to be up

worker

irflow-worker 23:44:29.50 Welcome to the Bitnami airflow-worker container
airflow-worker 23:44:29.50 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-worker 23:44:29.50 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-worker 23:44:29.50 
airflow-worker 23:44:29.51 INFO  ==> Enabling non-root system user with nss_wrapper
airflow-worker 23:44:29.51 INFO  ==> ** Starting Airflow setup **
airflow-worker 23:44:29.53 INFO  ==> No injected configuration file found. Creating default config file
airflow-worker 23:44:30.56 INFO  ==> Configuring Airflow database
airflow-worker 23:44:30.59 INFO  ==> Configuring Celery Executor
airflow-worker 23:44:30.61 INFO  ==> Trying to connect to the database server
airflow-worker 23:44:31.75 INFO  ==> Waiting for Airflow Webserver to be up
kwang1971 commented 11 months ago 
airflow-scheduler 02:11:19.99 
airflow-scheduler 02:11:20.00 Welcome to the Bitnami airflow-scheduler container
airflow-scheduler 02:11:20.00 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-scheduler 02:11:20.00 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-scheduler 02:11:20.00
airflow-scheduler 02:11:20.00 INFO  ==> Enabling non-root system user with nss_wrapper
airflow-scheduler 02:11:20.01 INFO  ==> ** Starting Airflow setup **
airflow-scheduler 02:11:20.03 INFO  ==> No injected configuration file found. Creating default config file
airflow-scheduler 02:11:20.56 INFO  ==> Configuring Airflow database
airflow-scheduler 02:11:20.58 INFO  ==> Configuring Celery Executor
airflow-scheduler 02:11:20.59 INFO  ==> Trying to connect to the database server
airflow-scheduler 02:11:21.66 INFO  ==> Waiting for Airflow Webserver to be up
timeout reached before the port went into state "inuse"
timeout reached before the port went into state "inuse"
kwang1971 commented 11 months ago
image
carrodher commented 11 months ago

Could you please try adding the BITNAMI_DEBUG=true to the containers to see if there is more info in the initialization logs?

kwang1971 commented 11 months ago

after add BITNAMI_DEBUG=true in scheduler container 

airflow-scheduler 12:27:48.20 
airflow-scheduler 12:27:48.21 Welcome to the Bitnami airflow-scheduler container
airflow-scheduler 12:27:48.21 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-scheduler 12:27:48.21 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-scheduler 12:27:48.21 
airflow-scheduler 12:27:48.22 INFO  ==> Enabling non-root system user with nss_wrapper
airflow-scheduler 12:27:48.22 INFO  ==> ** Starting Airflow setup **
airflow-scheduler 12:27:48.25 INFO  ==> No injected configuration file found. Creating default config file
2.7.1
airflow-scheduler 12:27:49.21 INFO  ==> Configuring Airflow database
airflow-scheduler 12:27:49.22 INFO  ==> Configuring Celery Executor
airflow-scheduler 12:27:49.23 INFO  ==> Trying to connect to the database server
[2023-09-29T12:27:50.216+0000] {db.py:1758} INFO - Connection successful.
airflow-scheduler 12:27:50.28 INFO  ==> Waiting for Airflow Webserver to be up
timeout reached before the port went into state "inuse"
timeout reached before the port went into state "inuse"
timeout reached before the port went into state "inuse"

worker container log:

airflow-worker 12:27:47.53 
airflow-worker 12:27:47.53 Welcome to the Bitnami airflow-worker container
airflow-worker 12:27:47.53 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-worker 12:27:47.53 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-worker 12:27:47.53 
airflow-worker 12:27:47.54 INFO  ==> Enabling non-root system user with nss_wrapper
airflow-worker 12:27:47.54 INFO  ==> ** Starting Airflow setup **
airflow-worker 12:27:47.56 INFO  ==> No injected configuration file found. Creating default config file
2.7.1
airflow-worker 12:27:48.65 INFO  ==> Configuring Airflow database
airflow-worker 12:27:48.67 INFO  ==> Configuring Celery Executor
airflow-worker 12:27:48.69 INFO  ==> Trying to connect to the database server
[2023-09-29T12:27:49.828+0000] {db.py:1758} INFO - Connection successful.
airflow-worker 12:27:49.92 INFO  ==> Waiting for Airflow Webserver to be up
timeout reached before the port went into state "inuse"
timeout reached before the port went into state "inuse"

airflow container

airflow 12:27:48.07 
airflow 12:27:48.08 Welcome to the Bitnami airflow container
airflow 12:27:48.08 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow 12:27:48.08 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow 12:27:48.08 
airflow 12:27:48.08 INFO  ==> Enabling non-root system user with nss_wrapper
airflow 12:27:48.09 INFO  ==> ** Starting Airflow setup **
airflow 12:27:48.10 INFO  ==> Validating settings in POSTGRESQL_CLIENT_* env vars
airflow 12:27:48.11 INFO  ==> Initializing Airflow ...
airflow 12:27:48.12 INFO  ==> No injected configuration file found. Creating default config file
airflow 12:27:51.28 INFO  ==> Configuring Airflow webserver authentication
airflow 12:27:51.29 INFO  ==> Configuring Airflow database
airflow 12:27:51.30 INFO  ==> Configuring Celery Executor
airflow 12:27:51.31 INFO  ==> Trying to connect to the database server
[2023-09-29T12:27:52.277+0000] {db.py:1758} INFO - Connection successful.
airflow 12:27:53.76 INFO  ==> Upgrading database schema
/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/cli/commands/db_command.py:64 DeprecationWarning: `db upgrade` is deprecated. Use `db migrate` instead.
DB: postgresql+psycopg2://bn_airflow:***@postgresql:5432/bitnami_airflow
Performing upgrade to the metadata database postgresql+psycopg2://bn_airflow:***@postgresql:5432/bitnami_airflow
[2023-09-29T12:27:55.684+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-09-29T12:27:55.684+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-09-29T12:27:55.747+0000] {db.py:1622} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Database migrating done!
airflow 12:27:59.47 INFO  ==> ** Airflow setup finished! **

airflow 12:27:59.48 INFO  ==> ** Starting Airflow **
/opt/bitnami/airflow/venv/lib/python3.9/site-packages/flask_limiter/extension.py:336 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
[2023-09-29 12:28:03 +0000] [119] [INFO] Starting gunicorn 21.2.0
[2023-09-29T12:28:06.113+0000] {providers_manager.py:271} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
[2023-09-29 12:28:07 +0000] [119] [INFO] Listening at: http://0.0.0.0:8080 (119)
[2023-09-29 12:28:07 +0000] [119] [INFO] Using worker: sync
[2023-09-29 12:28:07 +0000] [138] [INFO] Booting worker with pid: 138
[2023-09-29 12:28:07 +0000] [139] [INFO] Booting worker with pid: 139
[2023-09-29 12:28:07 +0000] [140] [INFO] Booting worker with pid: 140
[2023-09-29 12:28:07 +0000] [141] [INFO] Booting worker with pid: 141

restart scheduler container:

airflow-scheduler 12:32:51.42
airflow-scheduler 12:32:51.42 Welcome to the Bitnami airflow-scheduler container
airflow-scheduler 12:32:51.42 Subscribe to project updates by watching https://github.com/bitnami/containers
airflow-scheduler 12:32:51.42 Submit issues and feature requests at https://github.com/bitnami/containers/issues
airflow-scheduler 12:32:51.42
airflow-scheduler 12:32:51.42 INFO  ==> Enabling non-root system user with nss_wrapper
airflow-scheduler 12:32:51.42 INFO  ==> ** Starting Airflow setup **
airflow-scheduler 12:32:51.45 INFO  ==> Configuration file found, loading configuration
airflow-scheduler 12:32:51.45 INFO  ==> Trying to connect to the database server
[2023-09-29T12:32:52.156+0000] {db.py:1758} INFO - Connection successful.
airflow-scheduler 12:32:52.23 INFO  ==> Waiting for Airflow Webserver to be up
timeout reached before the port went into state "inuse"
carrodher commented 11 months ago

We are not able to reproduce the issue on our side. This seems to be a unique use case that might be challenging to replicate on our end, as it appears closely tied to your particular scenario.

For inquiries about the application itself, customizing its content, or questions about technology and infrastructure usage, we highly recommend referring to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

kwang1971 commented 11 months ago

Wrong setting in docker-compose.yml for Persisting at site https://hub.docker.com/r/bitnami/airflow: Need to add environment parameter in scheduler and work:- AIRFLOW_WEBSERVER_HOST=airflow

github-actions[bot] commented 11 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 11 months ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.