Task get failed. WooeyJob matching query does not exist

[adnan-alam]

We are using celery with redis to run the tasks. For the first few times the tasks run successfully no error shown on the console but after few submission we get the error shown below:

[2019-10-13 11:04:39,071: INFO/MainProcess] Connected to redis://localhost:6379//
[2019-10-13 11:04:39,081: INFO/MainProcess] mingle: searching for neighbors
[2019-10-13 11:04:40,099: INFO/MainProcess] mingle: all alone
[2019-10-13 11:04:40,113: INFO/MainProcess] celery@alpha ready.
[2019-10-13 11:04:43,971: INFO/MainProcess] Received task: wooey.tasks.submit_script[19f0f267-62a3-4dd3-ba35-7885f761a11f]  
[2019-10-13 11:04:43,990: ERROR/ForkPoolWorker-4] Signal handler <function task_completed at 0x7f518cd3bb70> raised: DoesNotExist('WooeyJob matching query does not exist.',)
Traceback (most recent call last):
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/celery/utils/dispatch/signal.py", line 288, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/dreamcat/Datagrabber_wooey/.env/src/wooey/wooey/signals.py", line 24, in task_completed
    job = WooeyJob.objects.get(pk=job_id)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/django/db/models/query.py", line 399, in get
    self.model._meta.object_name
wooey.models.core.WooeyJob.DoesNotExist: WooeyJob matching query does not exist.
[2019-10-13 11:04:43,994: ERROR/ForkPoolWorker-4] Task wooey.tasks.submit_script[19f0f267-62a3-4dd3-ba35-7885f761a11f] raised unexpected: DoesNotExist('WooeyJob matching query does not exist.',)
Traceback (most recent call last):
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/dreamcat/Datagrabber_wooey/.env/src/wooey/wooey/tasks.py", line 111, in submit_script
    job = WooeyJob.objects.get(pk=job_id)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/django/db/models/query.py", line 399, in get
    self.model._meta.object_name
wooey.models.core.WooeyJob.DoesNotExist: WooeyJob matching query does not exist.
[2019-10-13 11:04:43,995: ERROR/ForkPoolWorker-4] Signal handler <function task_completed at 0x7f518cd3bb70> raised: DoesNotExist('WooeyJob matching query does not exist.',)
Traceback (most recent call last):
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/celery/utils/dispatch/signal.py", line 288, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/dreamcat/Datagrabber_wooey/.env/src/wooey/wooey/signals.py", line 24, in task_completed
    job = WooeyJob.objects.get(pk=job_id)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/dreamcat/Datagrabber_wooey/.env/lib/python3.6/site-packages/django/db/models/query.py", line 399, in get
    self.model._meta.object_name
wooey.models.core.WooeyJob.DoesNotExist: WooeyJob matching query does not exist.

and task status keep showing waiting or error.

Sometimes Re run make the task run successfully but sometimes it fails. and Re submit , Clone also initiate this error.

This is the settings we are using in user_settings.py for celery and redis

INSTALLED_APPS += (
    'django_celery_results',
    'kombu.transport.filesystem',
)

# Celery settings
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'

broker_dir = ensure_path(os.path.join(BASE_DIR, '.broker'))
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "data_folder_in": ensure_path(os.path.join(broker_dir, "out")),
    "data_folder_out": ensure_path(os.path.join(broker_dir, "out")),
    "data_folder_processed": ensure_path(os.path.join(broker_dir, "processed")),
}

CELERY_TRACK_STARTED = True
WOOEY_CELERY = True
CELERY_SEND_EVENTS = True
CELERY_IMPORTS = ('wooey.tasks',)

# A cache interface. This provides realtime updates for scripts
WOOEY_REALTIME_CACHE = 'default'
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://localhost:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient"
        },
        "KEY_PREFIX": "wooey_cache"
    }
}

FYI we are using PostgreSQL db. Can't figure out the problem. Please help

[Chris7]

My first guess is celery is beating the database transaction. Are you able to try installing from a branch? I just put a PR up that should resolve that provided that is what is happening

[adnan-alam]

Thanks a ton :) The update fixed the issue we have faced. @Chris7