search

Tivix / django-cron

Write cron business logic as a Python class and let this app do the rest! It enables Django projects to schedule cron tasks, tracks their success / failures, manages contention (via a cache) etc. Basically takes care of all the boring work for you :-)
www.tivix.com
MIT License
900 stars 193 forks source link

added setting to run all cronjobs in parallel #88

Open FrankMeyer opened 8 years ago

FrankMeyer commented 8 years ago

DJANGO_CRON_MULTITHREADED, default is False Each job is run in a single thread, this does not affect the check for temporal overlaps of the same cronjob

FrankMeyer commented 8 years ago

This is usefull in case of multiple tasks sheduled at the same time. Then they are started all at the exact sheduled time. Multithreading is also useful in case you have many different tasks that should run every minute or so. Those will then not be sheduled one after the other anymore.

tab-cmd commented 7 years ago

Allowing parallels runs

By deafult parallels runs are not allowed (for security reasons). However if you want enable them just add:

ALLOW_PARALLEL_RUNS = True in your CronJob class.

Note

Note this requires a caching framework to be installed, as per https://docs.djangoproject.com/en/dev/topics/cache/

If you wish to override which cache is used, put this in your settings file:

DJANGO_CRON_CACHE = 'cron_cache'

FrankMeyer commented 7 years ago

Hello Sci-Tab,

you have misunderstood me, I don't want to run one CronJob in parallel, I want to start all different CronJobs that are enabled in parallel. I have multiple different Jobs to do and the starting times will not be exact, because the starting time of a job requires the jobs before to have ended (see runcrons.py). That is pretty bad, so I have added that option to start each different CronJob in a differend thread. Disallowing parallel runs of one CronJob still works.

greetings, Frank

lauritzen commented 6 years ago

Seems like a good and simple solution. @sci-tab any chance it can be merged in?

msopko81 commented 6 years ago

I would suggest having a variable to limit the number of threads running at a time. I tried this solution an had issues since it was trying to run all 60 jobs/threads at the same time.

laixintao commented 6 years ago

@msopko81 What's your problem exactly? Maybe we can add another variable to settings to set the max threading numbers.

msopko81 commented 6 years ago

Well, if you look at #114, I explained the issues I had. The main issue I had with this was all of the jobs running at the same time with an sqlite DB for cache and all 60 jobs trying to run at the same time cause the jobs to wait a long time trying to get sqlite write access. I had since switched back to REDIS (less errors occurring) and it is no so much of an issue, but other may have similar issues I had and would like to limit the number of concurrent threads running at the same time.

lauritzen commented 6 years ago

I can't see that @msopko81 problem should be a blocker. For those special cases just keep the default config. Any update on when this will me merged in?

bradyemerson commented 4 years ago

I might suggest that instead of using a true/false flag, have a INT variable for the number of threads with a default of 1. You could theoretically avoid the fork in the code, and you wouldn't need two settings to capture the behaviors discussed in the comments (threads Y/N and num of threads).