celery / billiard

Multiprocessing Pool Extensions
Other
419 stars 252 forks source link

Pool callback raised exception: TypeError("from_db_value() missing 1 required positional argument: 'context'",) #316

Closed sbasu1303 closed 4 years ago

sbasu1303 commented 4 years ago

I am getting the below error, any help appreciated. Pool callback raised exception: TypeError("from_db_value() missing 1 required positional argument: 'context'",) However my tasks are running, still why the error should come, please help me to correct my project configs.

celery.py:

`from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mydemoproj.settings')

app = Celery('mydemoproj')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.update({
    'task_routes': ('mydemoproj.celery.TaskRouter',),

})

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

class TaskRouter:
    def route_for_task(self, task, *args, **kwargs):
        if ':' not in task:
            return {'queue': 'default'}

        namespace, _ = task.split(':')
        return {'queue': namespace}`

settings.py

`djcelery.setup_loader()
BROKER_URL = REDIS_HOST
CELERY_ACCEPT_CONTENT = ['application/json', 'pickle']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_TRACK_STARTED = True
CELERY_SEND_EVENTS = True
CELERYD_PREFETCH_MULTIPLIER = 1
CELERY_ACKS_LATE = True
CELERY_CONCURRENCY = 8
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('long_tasks', Exchange('long_tasks'), routing_key='long_tasks'),
    Queue('quick_tasks', Exchange('quick_tasks'), routing_key='quick_tasks'),
    Queue('maintenance', Exchange('maintenance'), routing_key='maintenance'),
)
CELERY_IGNORE_RESULT = False
CELERYD_TASK_SOFT_TIME_LIMIT = 60*60`

When the error is occurring? -If I and scheduling async task from a periodic task.

myTask.py

`
from celery.task import periodic_task
from celery.task import task
from celery.schedules import crontab
...
...
@periodic_task(run_every=crontab(hour='*', minute='00'), name='long_tasks:hourlyTask')
def the_hourly_periodic_task():
batch = 0
    for item in getItems():
        td = datetime.utcnow() + timedelta(minutes=batch*20)
        sTask = run_enrich.si(item)
        task = sTask.apply_async(eta=td, queue='long_tasks', routing_key='long_tasks')
        batch += 1

@task()
def run_enrich(item):
    do_enrich(item)
`

Python - 3.8.3 and 3.6.9(occurring for both) Django==3.0.2 celery==4.4.5 django-celery==3.3.1 django-celery-beat==2.0.0 kombu==4.6.10 billiard==3.6.3.0

Error: [2020-06-24 09:44:08,589: ERROR/MainProcess] Pool callback raised exception: TypeError("from_db_value() missing 1 required positional argument: 'context'",) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1796, in safe_apply_callback fun(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/celery/worker/request.py", line 528, in on_failure store_result=self.store_errors, File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 169, in mark_as_failure traceback=traceback, request=request) File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 443, in store_result request=request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/djcelery/backends/database.py", line 33, in _store_result traceback=traceback, children=self.current_task_children(request), File "/usr/local/lib/python3.6/dist-packages/djcelery/managers.py", line 47, in _inner return fun(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/djcelery/managers.py", line 189, in store_result 'meta': {'children': children}}) File "/usr/local/lib/python3.6/dist-packages/djcelery/managers.py", line 92, in update_or_create return get_queryset(self).update_or_create(**kwargs) File "/usr/local/lib/python3.6/dist-packages/djcelery/managers.py", line 75, in update_or_create obj, created = self.get_or_create(**kwargs) File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 559, in get_or_create return self.get(**kwargs), False File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 411, in get num = len(clone) File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 258, in __len__ self._fetch_all() File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 1261, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 74, in __iter__ for row in compiler.results_iter(results): File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 1096, in apply_converters value = converter(value, expression, connection) TypeError: from_db_value() missing 1 required positional argument: 'context'

auvipy commented 4 years ago

line 1096, in apply_converters value = converter(value, expression, connection) TypeError: from_db_value() missing 1 required positional argument: 'context' seem to be a django issue. can you try celery==4.4.6?

sbasu1303 commented 4 years ago

@auvipy Tried celery 4.4.6 getting same error. Also forgot to mention in pip install -r requirements.txt I am getting an error: ERROR: django-celery 3.3.1 has requirement celery<4.0,>=3.1.15, but you'll have celery 4.4.6 which is incompatible.

sbasu1303 commented 4 years ago

currently this is the only error:

[2020-06-24 19:30:06,966: WARNING/ForkPoolWorker-5] /usr/local/lib/python3.8/dist-packages/celery/app/trace.py:622: RuntimeWarning: Exception raised outside body: TypeError("from_db_value() missing 1 required positional argume
nt: 'context'"):
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 479, in trace_task
    mark_as_done(
  File "/usr/local/lib/python3.8/dist-packages/celery/backends/base.py", line 158, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.8/dist-packages/celery/backends/base.py", line 442, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/usr/local/lib/python3.8/dist-packages/djcelery/backends/database.py", line 31, in _store_result
    self.TaskModel._default_manager.store_result(
  File "/usr/local/lib/python3.8/dist-packages/djcelery/managers.py", line 47, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/djcelery/managers.py", line 185, in store_result
    return self.update_or_create(task_id=task_id,
  File "/usr/local/lib/python3.8/dist-packages/djcelery/managers.py", line 92, in update_or_create
    return get_queryset(self).update_or_create(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/djcelery/managers.py", line 75, in update_or_create
    obj, created = self.get_or_create(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 411, in get
    num = len(clone)
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 258, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1261, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 74, in __iter__
    for row in compiler.results_iter(results):
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1096, in apply_converters
    value = converter(value, expression, connection)
TypeError: from_db_value() missing 1 required positional argument: 'context'
sbasu1303 commented 4 years ago

Is it like no async task from periodic_task?

auvipy commented 4 years ago

you are using wrong package. try django-celery-beat and django-celery-results