search

teamclairvoyant / airflow-maintenance-dags

A series of DAGs/Workflows to help maintain the operation of Airflow
Apache License 2.0
1.66k stars 393 forks source link

ERROR - pickle data was truncated on TaskInstance: airflow-db-cleanup.cleanup_BaseXCom #88

Open artik00 opened 4 years ago

artik00 commented 4 years ago

When Running %s on host %s <TaskInstance: airflow-db-cleanup.cleanup_BaseXCom 2020-09-09T00:00:00+00:00 It runs for 1 minute and then fails with : [2020-09-10 00:04:11,000] {taskinstance.py:1150} ERROR - pickle data was truncated Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 113, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable return self.python_callable(*self.op_args, *self.op_kwargs) File "/usr/local/airflow/dags/maintenance/airflow-db-cleanup.py", line 289, in cleanup_function entries_to_delete = query.all() File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all return list(self) File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 101, in instances cursor.close() File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in exit exc_value, with_traceback=exctb, File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise raise exception File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 81, in instances rows = [proc(row) for row in fetch] File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 81, in rows = [proc(row) for row in fetch] File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 602, in _instance state.manager.dispatch.load(state, context) File "/usr/local/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 322, in call fn(args, **kw) File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/mapper.py", line 3378, in _event_on_load instrumenting_mapper._reconstructor(state.obj()) File "/usr/local/lib/python3.6/site-packages/airflow/models/xcom.py", line 72, in init_on_load self.value = pickle.loads(self.value) _pickle.UnpicklingError: pickle data was truncated [2020-09-10 00:04:11,005] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=airflow-db-cleanup, task_id=cleanup_BaseXCom, execution_date=20200909T000000, start_date=20200910T000320, end_date=20200910T000411 [2020-09-10 00:04:15,716] {local_task_job.py:102} INFO - Task exited with return code 1

Kolpinetz commented 1 year ago

Hello @artik00,

Have you found a solution to the problem?

artik00 commented 1 year ago

@Kolpinetz I think I did back then, but now I don't remember what it was. I left my previous work so cannot check the code base. I vaguely remember implementing some parts of the cleanup myself, or something like that. Sorry I cannot help more.