Celery v4.3.0 issue with datetime `can not serialize 'datetime.datetime' object`

[ntarocco]

Celery 4.3 introduces a AsyncResult.date_done field to store the date when the task is completed. This date is stored as a datetime.datetime Python object. When using msgpack as celery task serializer, default in Invenio, this breaks serialization.

As result, your tasks are successfully but an exception will be raised afterwards:

[2019-09-12 11:30:22,112: ERROR/MainProcess] Pool callback raised exception: EncodeError(TypeError("can not serialize 'datetime.datetime' object",),)
Traceback (most recent call last):
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 370, in pack
    return packb(s, use_bin_type=True)
  File ".../venv/lib/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 282, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 288, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 285, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 232, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 279, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'datetime.datetime' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../venv/lib/python3.6/site-packages/billiard/pool.py", line 1791, in safe_apply_callback
    fun(*args, **kwargs)
  File ".../venv/lib/python3.6/site-packages/celery/worker/request.py", line 371, in on_failure
    store_result=self.store_errors,
  File ".../venv/lib/python3.6/site-packages/celery/backends/base.py", line 160, in mark_as_failure
    traceback=traceback, request=request)
  File ".../venv/lib/python3.6/site-packages/celery/backends/base.py", line 342, in store_result
    request=request, **kwargs)
  File ".../venv/lib/python3.6/site-packages/celery/backends/base.py", line 716, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File ".../venv/lib/python3.6/site-packages/celery/backends/base.py", line 293, in encode
    _, _, payload = self._encode(data)
  File ".../venv/lib/python3.6/site-packages/celery/backends/base.py", line 297, in _encode
    return dumps(data, serializer=self.serializer)
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File ".../lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File ".../venv/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File ".../venv/lib/python3.6/site-packages/kombu/serialization.py", line 370, in pack
    return packb(s, use_bin_type=True)
  File ".../venv/lib/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 282, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 288, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 285, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 232, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 279, in msgpack._cmsgpack.Packer._pack
kombu.exceptions.EncodeError: can not serialize 'datetime.datetime' object

The issue has been fixed and it will be released hopefully in Celery v4.4.

Workaround: install Celery < 4.3