search

puckel / docker-airflow

Docker Apache Airflow
Apache License 2.0
3.77k stars 541 forks source link

No tasks logs when using docker-compose celery #522

Open weichea opened 4 years ago

weichea commented 4 years ago

I have a dag that fails when I used celery but it runs fine when I use local executor. There is no task log available.

image

weichea commented 4 years ago

Realised it is due to celery task failure. But I have got no idea why celery task failed.

Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 69, in execute_command close_fds=True, env=env) File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['airflow', 'run', 'default', 'task', '2020-03-18T04:42:39.712148+00:00', '--local', '--pool', 'default_pool', '-sd', '/usr/local/airflow/dags/default_dag.py']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task R = retval = fun(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__ return self.run(args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 74, in execute_command raise AirflowException('Celery command failed') airflow.exceptions.AirflowException: Celery command failed