Open pedro-cf opened 1 week ago
andrew-stein-sp
commented
1 week ago having the same issue with 2.10.1 in k8s, using the CeleryKubernetesExecutor.
Could this be related to the inheritance issue that was discussed in https://github.com/apache/airflow/issues/41891?
pedro-cf
commented
1 week ago Additionally I checked the logs directory for the dag_id/run_id and it's missing the respective task_id folder.
adriens
commented
2 days ago Having the same issue on 2.10.0 through a podman-compose
adriens
commented
2 days ago We have upgraded on 2.10.1 like @andrew-stein-sp and we could reproduce the same behavior
sosystems-dev
commented
1 day ago got the same behavior since upgrading from version 2.9.3 to 2.10.1. We are using LocalExecutor
mn7k
commented
15 hours ago I have the same issue with 2.10.0, using the CeleryExecutor. It worked before I upgrading from version 2.9.0 to 2.10.0.
*** Could not read served logs: Invalid URL 'http://:8793/log/dag_id=service_stop/run_id=manual__2024-09-18T09:42:54+09:00/task_id=make_accountlist_task/attempt=1.log': No host suppliedeventlog
Executor CeleryExecutor(parallelism=6) reported that the task instance <TaskInstance: service_stop.make_accountlist_task manual__2024-09-18T09:42:54+09:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externallyScheduler has a error log at the same hour as eventlog.
[2024-09-18T00:43:18.036+0000] {celery_executor.py:291} ERROR - Error sending Celery task: module 'redis' has no attribute 'client'
Celery Task ID: TaskInstanceKey(dag_id='service_stop', task_id='make_accountlist_task', run_id='manual__2024-09-18T09:42:54+09:00', try_number=1, map_index=-1)
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 220, in send_task_to_executor
result = task_to_run.apply_async(args=[command], queue=queue)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/task.py", line 594, in apply_async
return app.send_task(
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 797, in send_task
with self.producer_or_acquire(producer) as P:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 932, in producer_or_acquire
producer, self.producer_pool.acquire, block=True,
^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 1354, in producer_pool
return self.amqp.producer_pool
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/amqp.py", line 591, in producer_pool
self.app.connection_for_write()]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 829, in connection_for_write
return self._connection(url or self.conf.broker_write_url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 880, in _connection
return self.amqp.Connection(
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kombu/connection.py", line 201, in __init__
if not get_transport_cls(transport).can_parse_url:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/__init__.py", line 91, in get_transport_cls
_transport_cache[transport] = resolve_transport(transport)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/__init__.py", line 76, in resolve_transport
return symbol_by_name(transport)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 282, in <module>
class PrefixedRedisPipeline(GlobalKeyPrefixMixin, redis.client.Pipeline):
^^^^^^^^^^^^
AttributeError: module 'redis' has no attribute 'client'
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.10.1
What happened?
I'm having an issue with an airflow instance where a task fails and I cannot read the logs.
Logs:
Event logs:
Additionally I checked the logs directory for the dag_id/run_id and it's missing the respective task_id folder.
What you think should happen instead?
I should be able to access the logs.
How to reproduce
Not sure how to.
Operating System
Ubuntu 24.04 LTS
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
Deployed with docker-compose on Docker Swarm setup on 2 VMs.
Anything else?
Additionally I checked the logs directory for the
dag_id/run_idand it's missing the respectivetask_idfolder.Are you willing to submit PR?
Code of Conduct