sibson / redbeat

RedBeat is a Celery Beat Scheduler that stores the scheduled tasks and runtime metadata in Redis.
Apache License 2.0
916 stars 130 forks source link

What happens to tasks when redis goes down or you wat to get bigger redis server? #220

Closed thisiseddy-ab closed 1 year ago

thisiseddy-ab commented 2 years ago

I'm trying to build a Podcast Listener,and want to check with Celery Cron Periodic Tasks when a new episode is Published.

I want to have the ability to add new Periodic Tasks with REST API, not jut from the go when u create the app.

My Question is what happens when redis server goes down, or i want a bigger redis server to handle more tasks ? From my understanding the tasks i created with REST API will be lost, only tasks that i created on the go will still be available.

I was using celery-sqlalchemy-scheduler but the library is not supported anymore, and has issues with postgress.

What should i do, should i save all tasks i create with redbeat in a database, and every time i restart redis or get bigger server i need check all tasks in postgress database and see if the exists in redis and created if does not exists ?

evanstjabadi commented 1 year ago

@thisiseddy-ab - would you still be interested if https://github.com/AngelLiang/celery-sqlalchemy-scheduler was forked and supported?

sibson commented 1 year ago

Either you need to use one of the HA/cluster redis modes, have a way to re-hydrate the tasks or trust the redis persistence