search

Betterment / delayed

a multi-threaded, SQL-driven ActiveJob backend used at Betterment to process millions of background jobs per day
MIT License
156 stars 9 forks source link

Periodic (Cron) jobs #17

Closed AxelTheGerman closed 2 years ago

AxelTheGerman commented 2 years ago

Hi and thank you for open sourcing this job framework!

I'm just starting to try it out but it looks quite promising. My mine motivator is the SQLite support.

One feature I'm missing (or haven't found yet) are periodic aka cron jobs. I have many use cases for some daily/weekly etc sync or update jobs.

Not sure if you think that should be part of delayed or are you maybe combining delayed with some kind of scheduler that runs alongside and simply queues jobs when their time has come? If so, any recommendations?

hknaksu commented 2 years ago

You may create rake tasks calling the jobs and invoke them using cron.

AxelTheGerman commented 2 years ago

I am hoping to avoid OS level integration, e.g. via cron directly.

But I think it makes sense to look for a separate scheduler which then queues jobs - it's just some job frameworks have taken on that complexity themselves (which has pros and cons of course)

hknaksu commented 2 years ago

take a look at whenever gem. it's for writing and deploying cron jobs.

jmileham commented 2 years ago

Betterment has done a fair amount of knowledge crunching on job scheduling as well (especially around as-close-to-once enqueue as possible), which as you note is not a core feature of delayed itself. @smudge may be able to drop some thoughts, but he's at RailsConf this week and might be a little busy!

AxelTheGerman commented 2 years ago

Finally RailsConf in person again, enjoy @smudge !

Yes that'd be interesting. especially around "as-close-to-once enqueue as possible"

AxelTheGerman commented 2 years ago

Going to have a look at crono too though it doesn't seem to be maintained anymore

https://github.com/plashchynski/crono

jmileham commented 2 years ago

Betterment's not-so-secret sauce here is a more opinionated fork of clockwork (which is a public repo but not documented or tidied up for external onboarding - maybe the diff is revealing) that implements DB-backed tracking of clock ticks to ensure we don't fire off the same work twice (either due to multiple processes or clock drift on a revived process) and logging a warning when we're behind. Unlike crono, the scheduled job definitions are just in source code - we removed the optional DB-backed job feature of clockwork - we like source code for change management. We've got a bunch of practices we deploy to make scheduled work safe by default, including negative alerting on key jobs. We should put some time into gussying it up for consumption. There are also other patterns for repetitive work and bulk work in various stages of maturity at Betterment as well that lean on delayed for job enqueue and execution.

AxelTheGerman commented 2 years ago

Awesome thanks will have a look at that... came across clockwork too.

For now I actually use Dokku Docker Local Scheduler which is cron based as that's my deployment infrastructure.

But I think for anyone looking into that this discussion should help and maybe the DB backend for clockwork can be an option in the future - that diff is definitely super helpful.

Happy to discuss further and lend a hand, but also okay closing this issue for now.

Thanks for all your time and work!