celerybeat redis ignores CELERYBEAT_SCHEDULE setting

[szelga]

$ celery -A apps.myproject beat -l info
celery beat v3.1.17 (Cipater) is starting.
Configuration ->
    . broker -> redis://localhost:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> now (0s)
[2014-12-23 20:30:47,539: INFO/MainProcess] beat: Starting...
[2014-12-23 20:30:57,690: INFO/MainProcess] Scheduler: Sending due task update-tickets-index (helpdesk.tasks.update_tickets_index)
[2014-12-23 20:31:07,713: INFO/MainProcess] Scheduler: Sending due task update-tickets-index (helpdesk.tasks.update_tickets_index)
[2014-12-23 20:31:17,714: INFO/MainProcess] Scheduler: Sending due task update-tickets-index (helpdesk.tasks.update_tickets_index)
[2014-12-23 20:31:27,717: INFO/MainProcess] Scheduler: Sending due task update-tickets-index (helpdesk.tasks.update_tickets_index)

$ celery -A apps.myproject beat -l info -S celerybeatredis.schedulers.RedisScheduler
celery beat v3.1.17 (Cipater) is starting.
Configuration ->
    . broker -> redis://localhost:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celerybeatredis.schedulers.RedisScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> now (0s)
[2014-12-23 20:31:50,722: INFO/MainProcess] beat: Starting...
[2014-12-23 20:31:50,723: INFO/MainProcess] backend scheduler using redis://localhost:6379/0

my configuration:

BROKER_URL = 'redis://localhost:6379/0'
BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}  # 1 hour.
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json',]
CELERY_REDIS_SCHEDULER_URL = 'redis://localhost:6379/0'
CELERY_TIMEZONE = TIME_ZONE

CELERYBEAT_SCHEDULE = {
    'update-tickets-index': {
        'task': 'helpdesk.tasks.update_tickets_index',
        'schedule': timedelta(seconds=10),
        'args': ()
    },
}

CELERY_REDIS_SCHEDULER_KEY_PREFIX = 'tasks:project:development:meta:'

related packages:

django-redis==3.7.2

celery==3.1.17
django-celery-transactions==0.2.0
celerybeat-redis==0.0.7

[szelga]

here's the debug output:

$ celery -A apps.myproject beat -l debug -S celerybeatredis.schedulers.RedisScheduler    
celery beat v3.1.17 (Cipater) is starting.
__    -    ... __   -        _
Configuration ->
    . broker -> redis://localhost:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celerybeatredis.schedulers.RedisScheduler

    . logfile -> [stderr]@%DEBUG
    . maxinterval -> now (0s)
[2014-12-23 20:39:16,164: DEBUG/MainProcess] Setting default socket timeout to 30
[2014-12-23 20:39:16,164: INFO/MainProcess] beat: Starting...
[2014-12-23 20:39:16,164: INFO/MainProcess] backend scheduler using redis://localhost:6379/0
[2014-12-23 20:39:16,164: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2014-12-23 20:39:16,165: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.

why does it wake in 5 min.? is it some redis limitation?

[kongluoxing]

yep, it ignore CELERYBEAT_SCHEDULE，because you must make schedule in redis。CELERYBEAT_SCHEDULE only reads by the builtin scheduler. please read README, it explained how to set your scheduler.

[timfeirg]

Hi, I just encountered the same problem, by following the project readme.

I tried to start celery beat with celerybeatredis.schedulers.RedisScheduler but it didn't load the scheduled tasks in celeryconfig.py, but when I start the beat using build-in scheduler, it works.

This section says that celerybeat-redis will load CELERYBEAT_SCHEDULE into redis, should we delete it? @kongluoxing

starting celery beat using built-in scheduler

$ celery -A tackle.app:tackleapp beat --loglevel=DEBUG
celery beat v3.1.23 (Cipater) is starting.
__    -    ... __   -        _
Configuration ->
    . broker -> redis://someredis.net:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%DEBUG
    . maxinterval -> now (0s)
[2016-04-21 14:37:26,917: DEBUG/MainProcess] Setting default socket timeout to 30
[2016-04-21 14:37:26,918: INFO/MainProcess] beat: Starting...
[2016-04-21 14:37:26,935: DEBUG/MainProcess] Current schedule:
<Entry: monitor-eru-apps tackle.tasks.trigger_monitor_eru_apps() <crontab: * * * * * (m/h/d/dM/MY)>
<Entry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
[2016-04-21 14:37:26,936: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2016-04-21 14:37:26,973: DEBUG/MainProcess] beat: Waking up in 33.02 seconds.

starting celery beat using RedisScheduler

$ celery -A tackle.app:tackleapp beat -S celerybeatredis.schedulers.RedisScheduler --loglevel=DEBUG
celery beat v3.1.23 (Cipater) is starting.
__    -    ... __   -        _
Configuration ->
    . broker -> redis://someredis:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celerybeatredis.schedulers.RedisScheduler
    . db -> redis://someredis.net
    . logfile -> [stderr]@%DEBUG
    . maxinterval -> now (0s)
[2016-04-21 14:37:35,055: DEBUG/MainProcess] Setting default socket timeout to 30
[2016-04-21 14:37:35,055: INFO/MainProcess] beat: Starting...
[2016-04-21 14:37:35,055: INFO/MainProcess] backend scheduler using redis://someredis.net
[2016-04-21 14:37:35,062: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2016-04-21 14:37:35,062: DEBUG/MainProcess] RedisScheduler: Fetching database schedule
[2016-04-21 14:37:35,064: DEBUG/MainProcess] DB schedule : {}
[2016-04-21 14:37:35,064: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
^C[2016-04-21 14:37:35,752: INFO/MainProcess] Writing modified entries...
[2016-04-21 14:37:35,752: INFO/MainProcess] Writing modified entries...

[kongluoxing]

I'll check this, RedisScheduler should load entries from celeryconfig

[rhockenbury]

@kongluoxing

Can this issue be closed with the merge of #16?