jie8357IOII commented 4 years ago

Checklist

[x] I have verified that the issue exists against the master branch of Celery.
[ ] This has already been asked to the discussion group first.
[x] I have read the relevant section in the contribution guide on reporting bugs.
[x] I have checked the issues list for similar or identical bug reports.
[x] I have checked the pull requests list for existing proposed fixes.
[x] I have checked the commit log to find out if the bug was already fixed in the master branch.
[x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

[x] I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
[x] I have verified that the issue exists against the master branch of Celery.
[ ] I have included the contents of pip freeze in the issue.
[ ] I have included all the versions of all the external dependencies required to reproduce this bug.

Optional Debugging Information

[ ] I have tried reproducing the issue on more than one Python version and/or implementation.
[ ] I have tried reproducing the issue on more than one message broker and/or result backend.
[ ] I have tried reproducing the issue on more than one version of the message broker and/or result backend.
[ ] I have tried reproducing the issue on more than one operating system.
[ ] I have tried reproducing the issue on more than one workers pool.
[ ] I have tried reproducing the issue with autoscaling, retries, ETA/Countdown & rate limits disabled.
[ ] I have tried reproducing the issue after downgrading and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

6258
6259

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

``` ```

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: N/A or Unknown
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

``` ```

Other Dependencies

N/A

Minimally Reproducible Test Case

```python ```

Expected Behavior

It will return a result when the task finished. And it should return the trace log or task state when the task failed.

Actual Behavior

When we try to get the failed task trace log or state, it will be failed because it tries to decode variable without encoding...

I think the following code is why the error happens.

File "/usr/local/lib/python3.6/site-packages/celery/backends/mongodb.py" in _get_task_meta_for
  209.                 'traceback': self.decode(obj['traceback']),

The full log is here.

Traceback:
File "/usr/local/lib/python3.6/json/decoder.py" in raw_decode
  355.             obj, end = self.scan_once(s, idx)

During handling of the above exception (0), another exception occurred:

File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py" in _reraise_errors
  50.         yield

File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py" in loads
  263.                     return decode(data)

File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py" in loads
  91.         return _loads(s)

File "/usr/local/lib/python3.6/json/__init__.py" in loads
  354.         return _default_decoder.decode(s)

File "/usr/local/lib/python3.6/json/decoder.py" in decode
  339.         obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/local/lib/python3.6/json/decoder.py" in raw_decode
  357.             raise JSONDecodeError("Expecting value", s, err.value) from None

During handling of the above exception (Expecting value: line 1 column 1 (char 0)), another exception occurred:

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  115.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  113.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/usr/local/lib/python3.6/site-packages/rest_framework/viewsets.py" in view
  114.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  505.             response = self.handle_exception(exc)

File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py" in handle_exception
  465.             self.raise_uncaught_exception(exc)

File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py" in raise_uncaught_exception
  476.         raise exc

File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  502.             response = handler(request, *args, **kwargs)

File "/code/labeling/resources/autolabeling_manager.py" in auto_labeling
  119.                 result = chord(ninja_api_tasks)(callback_ninja_api).get()

File "/usr/local/lib/python3.6/site-packages/celery/result.py" in get
  237.             on_message=on_message,

File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py" in wait_for_pending
  664.             no_ack=no_ack,

File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py" in wait_for
  691.                 on_interval()

File "/usr/local/lib/python3.6/site-packages/vine/promises.py" in __call__
  170.                 return self.throw()

File "/usr/local/lib/python3.6/site-packages/vine/promises.py" in __call__
  167.                     retval = fun(*final_args, **final_kwargs)

File "/usr/local/lib/python3.6/site-packages/celery/result.py" in _maybe_reraise_parent_error
  243.             node.maybe_throw()

File "/usr/local/lib/python3.6/site-packages/celery/result.py" in maybe_throw
  616.             result.maybe_throw(callback=callback, propagate=propagate)

File "/usr/local/lib/python3.6/site-packages/celery/result.py" in maybe_throw
  338.         cache = self._get_task_meta() if self._cache is None else self._cache

File "/usr/local/lib/python3.6/site-packages/celery/result.py" in _get_task_meta
  423.             return self._maybe_set_cache(self.backend.get_task_meta(self.id))

File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py" in get_task_meta
  523.                 meta = self._get_task_meta_for(task_id)

File "/usr/local/lib/python3.6/site-packages/celery/backends/mongodb.py" in _get_task_meta_for
  209.                 'traceback': self.decode(obj['traceback']),

File "/usr/local/lib/python3.6/site-packages/celery/backends/mongodb.py" in decode
  183.         return super(MongoBackend, self).decode(data)

File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py" in decode
  353.                      accept=self.accept)

File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py" in loads
  263.                     return decode(data)

File "/usr/local/lib/python3.6/contextlib.py" in __exit__
  99.                 self.gen.throw(type, value, traceback)

File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py" in _reraise_errors
  54.         reraise(wrapper, wrapper(exc), sys.exc_info()[2])

File "/usr/local/lib/python3.6/site-packages/vine/five.py" in reraise
  194.             raise value.with_traceback(tb)

File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py" in _reraise_errors
  50.         yield

File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py" in loads
  263.                     return decode(data)

File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py" in loads
  91.         return _loads(s)

File "/usr/local/lib/python3.6/json/__init__.py" in loads
  354.         return _default_decoder.decode(s)

File "/usr/local/lib/python3.6/json/decoder.py" in decode
  339.         obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/local/lib/python3.6/json/decoder.py" in raw_decode
  357.             raise JSONDecodeError("Expecting value", s, err.value) from None

Exception Type: DecodeError at /api/operations/auto/labeling/
Exception Value: Expecting value: line 1 column 1 (char 0)

yuquant commented 3 years ago

I use celery==5.0.0, pymongo==3.11.2, mongodb==4.4.2 .And I just replace the code line 205 of celery/celery/backends/mongodb.py with

'traceback': obj['traceback'],

and it works now but I don't know whether it will have side effects.

thedrow commented 3 years ago

Please provide a reproducible test case. I can't help you without reproducing the issue myself.

yuquant commented 3 years ago

Please provide a reproducible test case. I can't help you without reproducing the issue myself.

Thanks for your attention.By the way, python==3.6.8 @thedrow

consumer tasks.py

from time import sleep
from celery import Celery

app = Celery('tasks', backend='mongodb://172.17.0.10:27017/celery_data', broker='pyamqp://172.17.0.11:5672',
             # include=['dist_tasks.tasks']
             )

@app.task
def sub(x: int, y: int):
    for i in range(10):
        sleep(0.5)
    return x - y

productor client_demo.py

from time import sleep
from celery import Celery, exceptions
app = Celery('tasks', backend='mongodb://172.17.0.10:27017/celery_data', broker='pyamqp://172.17.0.11:5672',)

def test_case():
    result = app.send_task('tasks.sub', kwargs={'x': 3, 'y': '4'})  
    # I change the y from int to string to call the exception
    sleep(2)
    # print(result.ready())
    print(result.id, result.status, result.result)
    ret = result.get(timeout=15)  # exception here

if __name__ == '__main__':
    test_case()

exception


Traceback (most recent call last):
  File "/mnt/venv4fastapi/lib/python3.6/site-packages/kombu/serialization.py", line 49, in _reraise_errors
    yield
  File "/mnt/venv4fastapi/lib/python3.6/site-packages/kombu/serialization.py", line 262, in loads
    return decode(data)
  File "/mnt/venv4fastapi/lib/python3.6/site-packages/kombu/utils/json.py", line 85, in loads
    return _loads(s)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)```

celery / celery

MongoDB result backend raise error message when try to get task traceback and state #6364