rq / rq-scheduler

A lightweight library that adds job scheduling capabilities to RQ (Redis Queue)
MIT License
1.45k stars 230 forks source link

Registering death automatically after some time "pickel data was truncated" #219

Closed dhamechavivek95 closed 4 years ago

dhamechavivek95 commented 4 years ago

I am using ubuntu 19.04, python 3.6, rq-scheduler 0.9.1, and Django 2.2.4.

I need to register the scheduler and for that, I am using django-extension.

From django-extension, I am adding data into my queue and running queue from rq-worker.

step 1: starting django server step 2: running rqscheduler step 3: running python3 manage.py runscript scheduler step 4: running specific worker rqworker MY_WORKER_NAME

My scheduler script is running every one minute scheduler.cron("* * * * *", func=my_func, queue_name=QueueName.My_QUEUE_NAME)

After running four to five minutes rqscheduler stopped and throws an error

19:23:59 Registering death
Traceback (most recent call last):
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq/job.py", line 53, in unpickle
    obj = loads(pickled_string)
_pickle.UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/edugem/apps/psh_core/venv/bin/rqscheduler", line 10, in <module>
    sys.exit(main())
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq_scheduler/scripts/rqscheduler.py", line 61, in main
    scheduler.run(burst=args.burst)
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq_scheduler/scheduler.py", line 413, in run
    self.enqueue_jobs()
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq_scheduler/scheduler.py", line 391, in enqueue_jobs
    for job in jobs:
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq_scheduler/scheduler.py", line 322, in get_jobs
    job = self.job_class.fetch(job_id, connection=self.connection)
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq/job.py", line 288, in fetch
    job.refresh()
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq/job.py", line 462, in refresh
    self.restore(data)
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq/job.py", line 435, in restore
    self._result = unpickle(obj.get('result')) if obj.get('result') else None  # noqa
  File "/opt/edugem/apps/psh_core/venv/lib/python3.7/site-packages/rq/job.py", line 55, in unpickle
    raise UnpickleError('Could not unpickle', pickled_string, e)
rq.exceptions.UnpickleError: ('Could not unpickle', UnpicklingError('pickle data was truncated'))

Now It only can able to re-run rqscheduler after clearing all my redis records. But need a genuine solution.