search

whole-tale / girder_wholetale

Girder plugin providing basic Whole Tale functionality
BSD 3-Clause "New" or "Revised" License
3 stars 5 forks source link

Chained async tasks result in error in celery worker #447

Open Xarthisius opened 3 years ago

Xarthisius commented 3 years ago

Steps to reproduce

  1. Try to launch any instance

Expected result

Everything runs

Actual results

From celery_worker logs:

[2021-02-08 21:48:11,942: ERROR/ForkPoolWorker-2] Task gwvolman.tasks.build_tale_image[f1787146-44a3-472b-8c52-468dc565e189] raised unexpected: AttributeError("'_IDProxy' object has no attribute 'apply_async'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: '_apply_async'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 474, in trace_task
    _chsig.apply_async(
  File "/usr/local/lib/python3.8/dist-packages/celery/canvas.py", line 224, in apply_async
    _apply = self._apply_async
  File "/usr/local/lib/python3.8/dist-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.8/dist-packages/celery/canvas.py", line 516, in _apply_async
    return self.type.apply_async
AttributeError: '_IDProxy' object has no attribute 'apply_async'

Notes

It's caused by a backward incompatible release of jsonpickle-1.5.1 that we happen to hit during wholetale/girder image build. Both jsonpickle-1.5.0 and jsonpickle-1.5.2 work just fine. (See https://github.com/jsonpickle/jsonpickle/issues/351)

For now, let's pin it to 1.5.2.

TODO

Create a reduced test case and figure out where does it belong upstream.