coleifer / huey

a little task queue for python
https://huey.readthedocs.io/
MIT License
5.22k stars 370 forks source link

TypeError: cannot serialize '_io.BufferedReader' object #352

Closed riquedev closed 6 years ago

riquedev commented 6 years ago

I'm trying to run a Task inside a View in Django however I'm getting this error in the title.

settings.py | INSTALLED_APPS and HUEY

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'drf_customtoken',
    'django.contrib.staticfiles',
    'rest_framework',
    'django_filters',
    'rest_framework.authtoken',
    'after_response',
    'rest_framework_filters',
    'huey.contrib.djhuey',
    'rest_framework_swagger'
]

HUEY = {
    'backend': 'huey.backends.sqlite_backend',  # required.
    'name': 'RQDevAPIqueue',
    'connection': {'location': 'RQDevAPI-queue.db'},
    'always_eager': False, 

    'consumer_options': {'workers': 4},
}

View:


    def create(self, request, *args, **kwargs):

        history = Types.APIHistory()
        history.title = "Bulk Page Insert"
        history.save()

        self.async_create(request, history, *args, **kwargs)

        return Types.Response({
            "message": "Saved to APIHistory, check status.",
            "id": history.id
        }, status=Types.status.HTTP_201_CREATED)

    @Types.task()
    def async_create(self, request, history: Types.APIHistory, *args, **kwargs):
        try:
            super(PagesView, self).create(request, *args, **kwargs)
            history.is_finished = True
        except Exception as ex:
            history.exception = str(ex)

        history.save()
        return True

Traceback:

Traceback (most recent call last):
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
    response = get_response(request)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\django\core\handlers\base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\django\core\handlers\base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\django\views\decorators\csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\rest_framework\viewsets.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\django\utils\decorators.py", line 45, in _wrapper
    return bound_method(*args, **kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\django\views\decorators\csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\rest_framework\views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\rest_framework\views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\rest_framework\views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\RQDevFacebookAPI\models\pages\views.py", line 50, in create
    self.async_create(request, history, *args, **kwargs)
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\huey\api.py", line 635, in __call__
    return self.huey.enqueue(self.s(*args, **kwargs))
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\huey\api.py", line 306, in enqueue
    self._enqueue(self.registry.get_message_for_task(task))
  File "E:\Desenvolvimento\BTB\RQDevAPI\venv\lib\site-packages\huey\registry.py", line 67, in get_message_for_task
    on_complete))
TypeError: cannot serialize '_io.BufferedReader' object

Where am I going wrong?

coleifer commented 6 years ago

Huey pickles the task arguments because it has to serialize them and send them to the workers. Some objects cannot be pickled -- the io.BufferedReader being one of them (for what I hope are obvious reasons).

In other words, you cannot pass a request to a huey worker, which this error message is telling you.