apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.71k stars 14.21k forks source link

Airflow Scheduler Getting fail #31090

Closed taha-alphavu closed 1 year ago

taha-alphavu commented 1 year ago

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow Scheduler is getting failed randomly and after checking the server I have got this issue. We had Reset and initialize the Database but still same error. Can anyone help us out in this error?

[2023-05-05 12:23:18,597] {__init__.py:56} WARNING - Failed operation _get_task_meta_for.  Retrying 2 more times.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ProtocolViolation: invalid string in message

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 51, in _inner
    return fun(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 153, in _get_task_meta_for
    task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3389, in __iter__
    return self._execute_and_instances(context)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
    distilled_params,
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (psycopg2.errors.ProtocolViolation) invalid string in message

[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback 
FROM celery_taskmeta 
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': '2652a5ec-f13f-455b-9aab-0e98698b75c6'}]
(Background on this error at: http://sqlalche.me/e/4xp6)

What you think should happen instead

No response

How to reproduce

We had tried resolving it through resetting and initializing database but still could not found anything.

Operating System

Ubuntu 15.04

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.