Open potiuk opened 1 year ago
comment: Seems that indeed we have a circular import happening when UIAlert is created in local_settins.
Very interesting bug :) .
The root cause is AIP-51 related change https://github.com/apache/airflow/pull/28375 which added executor initialization in Sentry's ConfiguredSentry() __init__
class. This is done only in order to see if the executor supports sentry.
if executor_class.supports_sentry:
from sentry_sdk.integrations.celery import CeleryIntegration
sentry_celery = CeleryIntegration()
integrations.append(sentry_celery)
It's an interesting sequence of events:
@o-nikolas @rkarish you might want to take a look and do some other way of checking if the executor we have supports sentry - without necessarily initializing it. This might be tricky.
BTW. Initializing UI Alert view in local settings is perfectly fine and recommended by us in case anyone asked.
cc: @gil-tober who created the original issue.
As discussed in https://github.com/apache/airflow/discussions/31408#discussioncomment-5958636 - maybe a good solution could be utilizing listeners on_starting
to create the UIAlertView after the initialization is complete. I don't think we want to make it the "only" way though - local_settings is a more "natural" way of doing it, so possibly fixing this circular import might be a good idea anyway.
This is mentioned in #31408 Adding it here for anyone else having this issue
I will try the workaround you suggested and will post my findings.
For now, I found that setting _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1
also works as workaround.
Our metadb is Postgres, so the check for sqlite
is not relevant for us
I faced a similar issue today with Airflow 2.8.4 (Db migration worked successfully but later, the scheduler and webserver can not start and keep crashing due to the error below)
Our environment:
airflowLocalSettings
in our helm-chart values.yaml contains from airflow.www.utils import UIAlert
Error log from the scheduler
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Failed to import airflow_local_settings.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/airflow/settings.py", line 500, in import_local_settings
import airflow_local_settings
File "/opt/airflow/config/airflow_local_settings.py", line 2, in <module>
from airflow.www.utils import UIAlert
File "/usr/local/lib/python3.10/site-packages/airflow/www/utils.py", line 44, in <module>
from airflow.models.dagrun import DagRun
File "/usr/local/lib/python3.10/site-packages/airflow/models/dagrun.py", line 59, in <module>
from airflow.models.taskinstance import TaskInstance as TI
File "/usr/local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 99, in <module>
from airflow.sentry import Sentry
File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 198, in <module>
Sentry = ConfiguredSentry()
File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 90, in __init__
executor_class, _ = ExecutorLoader.import_default_executor_cls(validate=False)
File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 167, in import_default_executor_cls
executor, source = cls.import_executor_cls(executor_name, validate=validate)
File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 141, in import_executor_cls
return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 135, in _import_and_validate
executor = import_string(path)
File "/usr/local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 39, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 40, in <module>
from airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookException, PodReconciliationError
File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator.py", line 50, in <module>
from airflow.providers.cncf.kubernetes.pod_generator_deprecated import (
File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator_deprecated.py", line 33, in <module>
from airflow.utils.hashlib_wrapper import md5
File "/usr/local/lib/python3.10/site-packages/airflow/utils/hashlib_wrapper.py", line 26, in <module>
from airflow import PY39
ImportError: cannot import name 'PY39' from partially initialized module 'airflow' (most likely due to a circular import) (/usr/local/lib/python3.10/site-packages/airflow/__init__.py)
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/usr/local/lib/python3.10/site-packages/airflow/__init__.py", line 68, in <module>
settings.initialize()
File "/usr/local/lib/python3.10/site-packages/airflow/settings.py", line 552, in initialize
import_local_settings()
File "/usr/local/lib/python3.10/site-packages/airflow/settings.py", line 500, in import_local_settings
import airflow_local_settings
File "/opt/airflow/config/airflow_local_settings.py", line 2, in <module>
from airflow.www.utils import UIAlert
File "/usr/local/lib/python3.10/site-packages/airflow/www/utils.py", line 44, in <module>
from airflow.models.dagrun import DagRun
File "/usr/local/lib/python3.10/site-packages/airflow/models/dagrun.py", line 59, in <module>
from airflow.models.taskinstance import TaskInstance as TI
File "/usr/local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 99, in <module>
from airflow.sentry import Sentry
File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 198, in <module>
Sentry = ConfiguredSentry()
File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 90, in __init__
executor_class, _ = ExecutorLoader.import_default_executor_cls(validate=False)
File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 167, in import_default_executor_cls
executor, source = cls.import_executor_cls(executor_name, validate=validate)
File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 141, in import_executor_cls
return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 135, in _import_and_validate
executor = import_string(path)
File "/usr/local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 39, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 40, in <module>
from airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookException, PodReconciliationError
File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator.py", line 50, in <module>
from airflow.providers.cncf.kubernetes.pod_generator_deprecated import (
File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator_deprecated.py", line 33, in <module>
from airflow.utils.hashlib_wrapper import md5
File "/usr/local/lib/python3.10/site-packages/airflow/utils/hashlib_wrapper.py", line 26, in <module>
from airflow import PY39
ImportError: cannot import name 'PY39' from partially initialized module 'airflow' (most likely due to a circular import) (/usr/local/lib/python3.10/site-packages/airflow/__init__.py)
Setting _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1
does not work.
I faced a similar issue today with Airflow 2.8.4 (Db migration worked successfully but later, the scheduler and webserver can not start and keep crashing due to the error below)
This is a different issue and likely to be addressed by https://github.com/apache/airflow/pull/39062 - which should be available in the next cncf.kubernetes after merging.
Discussed in https://github.com/apache/airflow/discussions/31408