Multiple CronJobBase running when ALLOW_PARALLEL_RUNS is set to False

[eriktelepovsky]

Hi, first of all, thank you for your work. It is very useful app. I use it in many projects, but I came into problem in the last one. I always set ALLOW_PARALLEL_RUNS to False in all of my CronJobBase classes, but runcrons command always run long running tasks multiple times for me and server resources get overloaded in few minutes.

This is my cron job task configuration:

class LongRunningTask(CronJobBase):
    ALLOW_PARALLEL_RUNS = False
    RUN_EVERY_MINS = 30  # every 30 minutes
    RETRY_AFTER_FAILURE_MINS =  30
    MIN_NUM_FAILURES = 1
    schedule = Schedule(
        run_every_mins=RUN_EVERY_MINS, 
        retry_after_failure_mins=RETRY_AFTER_FAILURE_MINS
    )

crontab is set to check cron tasks every minute: */1 * * * * python manage.py runcrons;

I also tried to run runcrons for specific task as well, same result: */1 * * * * python manage.py runcrons myapp.cron.LongRunningTask;

Is there something wrong with my ALLOW_PARALLEL_RUNS configuration? Or is there some another option to disable multiple task running?

I tried versions 0.3.5 and 0.3.3. Same behaviour.

Thank you very much for your help in advance.

[craiglabenz]

Make sure you have a caching layer set up and configured, as that's where the app maintains state of which jobs are currently in-progress. Without that, the feature will silently fail and re-trigger the job every time the crontab comes up.

[eriktelepovsky]

Thank you for your answer. I had set default Django cache, which is django.core.cache.backends.locmem.LocMemCache. I changed it to django.core.cache.backends.filebased.FileBasedCache and it started working. It is because LocMemCache is per-process (according to documentation). Each process has its own private cache instance, which means no cross-process caching is possible. That's the reason why the cronjobs couldn't see each other locks and they run every time. It would be helpful to mention that in README or documentation. Thank you for your help.

[fish-ball]

@eriktelepovsky works like a charm.