Closed jape-dev closed 4 years ago
Hi,
Did you make any modifications to CELERYBEAT_SCHEDULE
, and did this originally work using the course's source code where the beat server was integrated with the worker using -B
?
Also in the future, can you use the course platform's discussion board or email me directly for support on the course? It's just easier to keep track of things, and others will be able to check out the answer to help avoid duplicates. Not too many folks use github issues for reporting this course's issues since the app in this repo isn't the same in the course.
Hi @nickjj ,
Yes, I've left the CELERYBEAT_SCHEDULE
config as is. Yes, I originally tried using the integrated approach with the -B
arg. I get a similar traceback:
File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
celery_1 | return obj.__dict__[self.__name__]
celery_1 | KeyError: 'scheduler'
celery_1 |
celery_1 | During handling of the above exception, another exception occurred:
celery_1 |
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 485, in setup_schedule
celery_1 | self._store = self._open_schedule()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 475, in _open_schedule
celery_1 | return self.persistence.open(self.schedule_filename, writeback=True)
celery_1 | File "/usr/local/lib/python3.7/shelve.py", line 243, in open
celery_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
celery_1 | File "/usr/local/lib/python3.7/shelve.py", line 227, in __init__
celery_1 | Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
celery_1 | File "/usr/local/lib/python3.7/dbm/__init__.py", line 94, in open
celery_1 | return mod.open(file, flag, mode)
celery_1 | _gdbm.error: [Errno 22] Invalid argument
celery_1 | [2020-05-24 20:40:38,998: ERROR/Beat] Process Beat
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
celery_1 | return obj.__dict__[self.__name__]
celery_1 | KeyError: 'scheduler'
celery_1 |
celery_1 | During handling of the above exception, another exception occurred:
celery_1 |
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 485, in setup_schedule
celery_1 | self._store = self._open_schedule()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 475, in _open_schedule
celery_1 | return self.persistence.open(self.schedule_filename, writeback=True)
celery_1 | File "/usr/local/lib/python3.7/shelve.py", line 243, in open
celery_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
celery_1 | File "/usr/local/lib/python3.7/shelve.py", line 227, in __init__
celery_1 | Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
celery_1 | File "/usr/local/lib/python3.7/dbm/__init__.py", line 94, in open
celery_1 | return mod.open(file, flag, mode)
celery_1 | _gdbm.error: [Errno 22] Invalid argument
celery_1 |
celery_1 | During handling of the above exception, another exception occurred:
celery_1 |
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/billiard/process.py", line 327, in _bootstrap
celery_1 | self.run()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 673, in run
celery_1 | self.service.start(embedded_process=True)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 588, in start
celery_1 | humanize_seconds(self.scheduler.max_interval))
celery_1 | File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
celery_1 | value = obj.__dict__[self.__name__] = self.__get(obj)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 632, in scheduler
celery_1 | return self.get_scheduler()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 627, in get_scheduler
celery_1 | lazy=lazy,
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 467, in __init__
celery_1 | Scheduler.__init__(self, *args, **kwargs)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 226, in __init__
celery_1 | self.setup_schedule()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 493, in setup_schedule
celery_1 | self._store = self._destroy_open_corrupted_schedule(exc)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 481, in _destroy_open_corrupted_schedule
celery_1 | return self._open_schedule()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 475, in _open_schedule
celery_1 | return self.persistence.open(self.schedule_filename, writeback=True)
celery_1 | File "/usr/local/lib/python3.7/shelve.py", line 243, in open
celery_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
celery_1 | File "/usr/local/lib/python3.7/shelve.py", line 227, in __init__
celery_1 | Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
celery_1 | File "/usr/local/lib/python3.7/dbm/__init__.py", line 94, in open
celery_1 | return mod.open(file, flag, mode)
celery_1 | _gdbm.error: [Errno 22] Invalid argument
celery_1 | [2020-05-24 20:40:39,002: WARNING/Beat] Process Beat:
celery_1 | [2020-05-24 20:40:39,008: WARNING/Beat] Traceback (most recent call last):
celery_1 | [2020-05-24 20:40:39,009: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
celery_1 | return obj.__dict__[self.__name__]
celery_1 | [2020-05-24 20:40:39,010: WARNING/Beat] KeyError: 'scheduler'
celery_1 | [2020-05-24 20:40:39,010: WARNING/Beat] During handling of the above exception, another exception occurred:
celery_1 | [2020-05-24 20:40:39,011: WARNING/Beat] Traceback (most recent call last):
celery_1 | [2020-05-24 20:40:39,013: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 485, in setup_schedule
celery_1 | self._store = self._open_schedule()
celery_1 | [2020-05-24 20:40:39,014: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 475, in _open_schedule
celery_1 | return self.persistence.open(self.schedule_filename, writeback=True)
celery_1 | [2020-05-24 20:40:39,016: WARNING/Beat] File "/usr/local/lib/python3.7/shelve.py", line 243, in open
celery_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
celery_1 | [2020-05-24 20:40:39,017: WARNING/Beat] File "/usr/local/lib/python3.7/shelve.py", line 227, in __init__
celery_1 | Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
celery_1 | [2020-05-24 20:40:39,017: WARNING/Beat] File "/usr/local/lib/python3.7/dbm/__init__.py", line 94, in open
celery_1 | return mod.open(file, flag, mode)
celery_1 | [2020-05-24 20:40:39,018: WARNING/Beat] _gdbm.error: [Errno 22] Invalid argument
celery_1 | [2020-05-24 20:40:39,019: WARNING/Beat] During handling of the above exception, another exception occurred:
celery_1 | [2020-05-24 20:40:39,020: WARNING/Beat] Traceback (most recent call last):
celery_1 | [2020-05-24 20:40:39,021: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/billiard/process.py", line 327, in _bootstrap
celery_1 | self.run()
celery_1 | [2020-05-24 20:40:39,024: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 673, in run
celery_1 | self.service.start(embedded_process=True)
celery_1 | [2020-05-24 20:40:39,025: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 588, in start
celery_1 | humanize_seconds(self.scheduler.max_interval))
celery_1 | [2020-05-24 20:40:39,026: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
celery_1 | value = obj.__dict__[self.__name__] = self.__get(obj)
celery_1 | [2020-05-24 20:40:39,027: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 632, in scheduler
celery_1 | return self.get_scheduler()
celery_1 | [2020-05-24 20:40:39,028: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 627, in get_scheduler
celery_1 | lazy=lazy,
celery_1 | [2020-05-24 20:40:39,029: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 467, in __init__
celery_1 | Scheduler.__init__(self, *args, **kwargs)
celery_1 | [2020-05-24 20:40:39,030: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 226, in __init__
celery_1 | self.setup_schedule()
celery_1 | [2020-05-24 20:40:39,031: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 493, in setup_schedule
celery_1 | self._store = self._destroy_open_corrupted_schedule(exc)
celery_1 | [2020-05-24 20:40:39,032: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 481, in _destroy_open_corrupted_schedule
celery_1 | return self._open_schedule()
celery_1 | [2020-05-24 20:40:39,032: WARNING/Beat] File "/usr/local/lib/python3.7/site-packages/celery/beat.py", line 475, in _open_schedule
celery_1 | return self.persistence.open(self.schedule_filename, writeback=True)
celery_1 | [2020-05-24 20:40:39,033: WARNING/Beat] File "/usr/local/lib/python3.7/shelve.py", line 243, in open
celery_1 | return DbfilenameShelf(filename, flag, protocol, writeback)
celery_1 | [2020-05-24 20:40:39,034: WARNING/Beat] File "/usr/local/lib/python3.7/shelve.py", line 227, in __init__
celery_1 | Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
celery_1 | [2020-05-24 20:40:39,035: WARNING/Beat] File "/usr/local/lib/python3.7/dbm/__init__.py", line 94, in open
celery_1 | return mod.open(file, flag, mode)
Apologies - I'll be sure to use the more appropriate forums for these kind of questions next time round.
Which section of the course are you working on?
Also have you tried to do docker-compose down
, delete the old celerybeat-schedule
file and then docker-compose up
?
I'm working on section 20-internationalizaton-i18n. Yes I have tried to delete the celerybeat-schedule
file as I've seen that recommended in other related posts. However this doesn't seem to have worked as upon running docker-compose up
again a newcelerybeat-schedule
file gets created which seems to throw an error when it is loaded by shelve
as per the traceback.
It's worth mentioning that I didn't get this error when working on section 16 where the periodic tasks were first introduced.
I've gone back and rebuilt the application through the different course sections. It seems that I first start getting this issue in section 17-building-the-snake-eyes-game.
Did you do the docker-compose down
step?
Yep I did: docker-compose down
--> deletedcelerybeat-schedule
-->docker-compose up
and get the same error.
Have pulled down a fresh version of the course folder and gone into section 17 and getting the same error.
I'm not able to reproduce that here.
Here's what I did to test:
docker-compose build --no-cache
docker-compose down -v
docker-compose up
At this point, everything is up and running including Celery.
Step 4 ensures everything is built from scratch and step 5's -v
removes all named volumes (all PostgreSQL and Redis data will be purged).
If you perform the same workflow, what happens?
I've followed your workflow exactly and unfortunately still getting the same error. Even getting the error when repeating the same steps for section 16 now too.
I'm starting to think it could be something to do with my Docker Toolbox setup. I'm using the latest release:
Client:
Version: 19.03.1
API version: 1.40
Go version: go1.12.7
Git commit: 74b1e89e8a
Built: Wed Jul 31 15:18:18 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:28:45 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
https://github.com/docker/toolbox/releases
Are you aware of any complications that may arise from this version? Or is this part of the build only compatible with docker desktop?
It'll work with Docker Toolbox too. Lots of folks are using it in the course.
When was the last time you downloaded the zip file from the course?
As a hail mary move you could always try deleting the VM that Docker Toolbox created and letting Docker Toolbox create a new one for you the next time you launch the Docker Quickstart Terminal.
I just downloaded a fresh version of the course zip file today while I was trying out your workflow. Hence why I'm so confused and starting to think it's something to do with my Docker Toolbox 😅
I've tried docker-machine restart
but I'll give deleting the VM a go using docker-machine kill
Removing the docker machine with docker-machine rm
and letting docker toolbox create a new one still didn't work 😢 I've been doing some more digging and found a thread that suggested adding a celery-beat.conf
file may work. I'll give this a shot https://docs.celeryproject.org/en/stable/userguide/daemonizing.html#init-script-celerybeat
If you email me from the account you purchased the course on I'm open to doing a quick Google Hangouts or Zoom screenshare session. It might be faster than a bunch of back / forths here.
So far you're the only person who has reported this and thousands of people have completed the course based on the stats I see.
That would be great - just sent you an email. Thanks
Closing this here, as the issue isn't related to the code in this repo.
Just wanted to post this in case anyone finds this thread looking for a solution to the same issue.
From celery docs
Beat needs to store the last run times of the tasks in a local database file (named celerybeat-schedule by default), so it needs access to write in the current directory, or alternatively you can specify a custom location for this
For some reason my application cannot read the celerybeat-schedule file. But there is a work around to store the active schedule in Redis instead. There a few packages that allow you to do this. I tried RedBeat and it worked 🎉
I'm trying to test the periodic celery tasks. However, when I try to start the celery container I get the following error:
As per the course notes, I have tried to amend the docker-compose.yml to run the celery and beat worker independently:
However I still get the same issue. I have also tried to delete the celerybeat-schedule file which seems to be corrupted however upon running
docker-compose up
the file gets recreated which throws the same error.Below is the full trace back: