"celery -A myapp inspect scheduled" does not list periodic tasks

ShaheedHaque commented 6 years ago

I have some periodic tasks defined which show up fine in the Django admin, and run. However, when I use the "inspect scheduled" command, I never get anything.

$ celery -A myapp inspect scheduled
-> celery@baba: OK
    - empty -

Note that the tasks were created dynamically using the Django models for CrontabSchedule and PeriodicTask; so they are not specified in the config. Is there some config that I am missing? This is what the slightly redacted "inspect report" says:

 $ cat t.t
  -> celery@baba: OK

    software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.6.3
                billiard:3.5.0.3 py-amqp:2.2.2
    platform -> system:Linux arch:64bit, ELF imp:CPython
    loader   -> celery.loaders.app.AppLoader
    settings -> transport:amqp results:django-db

    ABSOLUTE_URL_OVERRIDES: {
     }
    ADMINS: []
    ALLOWED_HOSTS: ['*']
    APPEND_SLASH: True
    AUTHENTICATION_BACKENDS: ['django.contrib.auth.backends.ModelBackend']
    AUTH_PASSWORD_VALIDATORS: '********'
    AUTH_USER_MODEL: 'myapp.User'
    BASE_DIR: '/home/srhaque/dev/testdjango'
    CACHES: {
     'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
    CACHE_MIDDLEWARE_ALIAS: 'default'
    CACHE_MIDDLEWARE_KEY_PREFIX: '********'
    CACHE_MIDDLEWARE_SECONDS: 600
    CELERY_BROKER_URL: 'amqp://guest:********@localhost:5672//'
    CELERY_RESULT_BACKEND: 'django-db'
    CSRF_COOKIE_AGE: 31449600
    CSRF_COOKIE_DOMAIN: None
    CSRF_COOKIE_HTTPONLY: False
    CSRF_COOKIE_NAME: 'csrftoken'
    CSRF_COOKIE_PATH: '/'
    CSRF_COOKIE_SECURE: False
    CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'
    CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'
    CSRF_TRUSTED_ORIGINS: []
    CSRF_USE_SESSIONS: False
    DATABASES: {
        'default': {   'ATOMIC_REQUESTS': False,
                       'AUTOCOMMIT': True,
                       'CONN_MAX_AGE': 0,
                       'ENGINE': 'django.db.backends.postgresql',
                       'HOST': '127.0.0.1',
                       'NAME': 'foo',
                       'OPTIONS': {},
                       'PASSWORD': '********',
                       'PORT': '5432',
                       'TEST': {   'CHARSET': None,
                                   'COLLATION': None,
                                   'MIRROR': None,
                                   'NAME': None},
                       'TIME_ZONE': None,
                       'USER': 'dbcoreuser'}}
    DATABASE_ROUTERS: '********'
    DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440
    DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000
    DATETIME_FORMAT: 'N j, Y, P'
    DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',
     '%Y-%m-%d %H:%M:%S.%f',
     '%Y-%m-%d %H:%M',
     '%Y-%m-%d',
     '%m/%d/%Y %H:%M:%S',
     '%m/%d/%Y %H:%M:%S.%f',
     '%m/%d/%Y %H:%M',
     '%m/%d/%Y',
     '%m/%d/%y %H:%M:%S',
     '%m/%d/%y %H:%M:%S.%f',
     '%m/%d/%y %H:%M',
     '%m/%d/%y']
    DATE_FORMAT: 'N j, Y'
    DATE_INPUT_FORMATS: ['%Y-%m-%d',
     '%m/%d/%Y',
     '%m/%d/%y',
     '%b %d %Y',
     '%b %d, %Y',
     '%d %b %Y',
     '%d %b, %Y',
     '%B %d %Y',
     '%B %d, %Y',
     '%d %B %Y',
     '%d %B, %Y']
    DEBUG: True
    DEBUG_PROPAGATE_EXCEPTIONS: False
    DECIMAL_SEPARATOR: '.'
    DEFAULT_CHARSET: 'utf-8'
    DEFAULT_CONTENT_TYPE: 'text/html'
    DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'
    DEFAULT_FILE_STORAGE: 'django.core.files.storage.FileSystemStorage'
    DEFAULT_FROM_EMAIL: 'webmaster@localhost'
    DEFAULT_INDEX_TABLESPACE: ''
    DEFAULT_TABLESPACE: ''
    DISALLOWED_USER_AGENTS: []
    EMAIL_BACKEND: 'django.core.mail.backends.smtp.EmailBackend'
    EMAIL_HOST: 'localhost'
    EMAIL_HOST_PASSWORD: '********'
    EMAIL_HOST_USER: ''
    EMAIL_PORT: 25
    EMAIL_SSL_CERTFILE: None
    EMAIL_SSL_KEYFILE: '********'
    EMAIL_SUBJECT_PREFIX: '[Django] '
    EMAIL_TIMEOUT: None
    EMAIL_USE_LOCALTIME: False
    EMAIL_USE_SSL: False
    EMAIL_USE_TLS: False
    FILE_CHARSET: 'utf-8'
    FILE_UPLOAD_DIRECTORY_PERMISSIONS: None
    FILE_UPLOAD_HANDLERS: 
        ('django_excel.ExcelMemoryFileUploadHandler',
     'django_excel.TemporaryExcelFileUploadHandler')
    FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
    FILE_UPLOAD_PERMISSIONS: None
    FILE_UPLOAD_TEMP_DIR: None
    FIRST_DAY_OF_WEEK: 0
    FIXTURE_DIRS: []
    FORCE_SCRIPT_NAME: None
    FORMAT_MODULE_PATH: None
    FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'
    IGNORABLE_404_URLS: []
    INSTALLED_APPS: ['myapp.apps.MyappConfig',
     'django.contrib.admin',
     'django.contrib.auth',
     'django.contrib.contenttypes',
     'django.contrib.sessions',
     'django.contrib.messages',
     'django.contrib.staticfiles',
     'django.contrib.postgres',
     'django_jinja',
     'bootstrapform_jinja',
     'django_celery_beat',
     'django_celery_monitor',
     'django_celery_results']
    INTERNAL_IPS: []
    LANGUAGES: [('af', 'Afrikaans'),
     ('ar', 'Arabic'),
     ('ast', 'Asturian'),
     ('az', 'Azerbaijani'),
     ('bg', 'Bulgarian'),
     ('be', 'Belarusian'),
     ('bn', 'Bengali'),
     ('br', 'Breton'),
     ('bs', 'Bosnian'),
     ('ca', 'Catalan'),
     ('cs', 'Czech'),
     ('cy', 'Welsh'),
     ('da', 'Danish'),
     ('de', 'German'),
     ('dsb', 'Lower Sorbian'),
     ('el', 'Greek'),
     ('en', 'English'),
     ('en-au', 'Australian English'),
     ('en-gb', 'British English'),
     ('eo', 'Esperanto'),
     ('es', 'Spanish'),
     ('es-ar', 'Argentinian Spanish'),
     ('es-co', 'Colombian Spanish'),
     ('es-mx', 'Mexican Spanish'),
     ('es-ni', 'Nicaraguan Spanish'),
     ('es-ve', 'Venezuelan Spanish'),
     ('et', 'Estonian'),
     ('eu', 'Basque'),
     ('fa', 'Persian'),
     ('fi', 'Finnish'),
     ('fr', 'French'),
     ('fy', 'Frisian'),
     ('ga', 'Irish'),
     ('gd', 'Scottish Gaelic'),
     ('gl', 'Galician'),
     ('he', 'Hebrew'),
     ('hi', 'Hindi'),
     ('hr', 'Croatian'),
     ('hsb', 'Upper Sorbian'),
     ('hu', 'Hungarian'),
     ('ia', 'Interlingua'),
     ('id', 'Indonesian'),
     ('io', 'Ido'),
     ('is', 'Icelandic'),
     ('it', 'Italian'),
     ('ja', 'Japanese'),
     ('ka', 'Georgian'),
     ('kab', 'Kabyle'),
     ('kk', 'Kazakh'),
     ('km', 'Khmer'),
     ('kn', 'Kannada'),
     ('ko', 'Korean'),
     ('lb', 'Luxembourgish'),
     ('lt', 'Lithuanian'),
     ('lv', 'Latvian'),
     ('mk', 'Macedonian'),
     ('ml', 'Malayalam'),
     ('mn', 'Mongolian'),
     ('mr', 'Marathi'),
     ('my', 'Burmese'),
     ('nb', 'Norwegian Bokmål'),
     ('ne', 'Nepali'),
     ('nl', 'Dutch'),
     ('nn', 'Norwegian Nynorsk'),
     ('os', 'Ossetic'),
     ('pa', 'Punjabi'),
     ('pl', 'Polish'),
     ('pt', 'Portuguese'),
     ('pt-br', 'Brazilian Portuguese'),
     ('ro', 'Romanian'),
     ('ru', 'Russian'),
     ('sk', 'Slovak'),
     ('sl', 'Slovenian'),
     ('sq', 'Albanian'),
     ('sr', 'Serbian'),
     ('sr-latn', 'Serbian Latin'),
     ('sv', 'Swedish'),
     ('sw', 'Swahili'),
     ('ta', 'Tamil'),
     ('te', 'Telugu'),
     ('th', 'Thai'),
     ('tr', 'Turkish'),
     ('tt', 'Tatar'),
     ('udm', 'Udmurt'),
     ('uk', 'Ukrainian'),
     ('ur', 'Urdu'),
     ('vi', 'Vietnamese'),
     ('zh-hans', 'Simplified Chinese'),
     ('zh-hant', 'Traditional Chinese')]
    LANGUAGES_BIDI: ['he', 'ar', 'fa', 'ur']
    LANGUAGE_CODE: 'en-us'
    LANGUAGE_COOKIE_AGE: None
    LANGUAGE_COOKIE_DOMAIN: None
    LANGUAGE_COOKIE_NAME: 'django_language'
    LANGUAGE_COOKIE_PATH: '/'
    LOCALE_PATHS: []
    LOGGING: {
        'disable_existing_loggers': False,
        'formatters': {   'standard': {   'format': '%(asctime)s [%(levelname)s] '
                                                    '%(name)s: %(message)s'}},
        'handlers': {   'file': {   'backupCount': 10,
                                    'class': 'logging.handlers.RotatingFileHandler',
                                    'filename': '/home/srhaque/.local/share/myapp/django.log',
                                    'formatter': 'standard',
                                    'level': 'DEBUG',
                                    'maxBytes': 15728640}},
        'loggers': {'': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}},
        'version': 1}
    LOGGING_CONFIG: 'logging.config.dictConfig'
    LOGGING_DIR: '/home/srhaque/.local/share/myapp'
    LOGIN_REDIRECT_URL: '/accounts/profile/'
    LOGIN_URL: '/accounts/login/'
    LOGOUT_REDIRECT_URL: None
    MANAGERS: []
    MEDIA_ROOT: ''
    MEDIA_URL: ''
    MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'
    MIDDLEWARE: ['django.middleware.security.SecurityMiddleware',
     'django.contrib.sessions.middleware.SessionMiddleware',
     'django.middleware.common.CommonMiddleware',
     'django.middleware.csrf.CsrfViewMiddleware',
     'django.contrib.auth.middleware.AuthenticationMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware',
     'django.middleware.clickjacking.XFrameOptionsMiddleware']
    MIGRATION_MODULES: {
     }
    MONTH_DAY_FORMAT: 'F j'
    NUMBER_GROUPING: 0
    PASSWORD_HASHERS: '********'
    PASSWORD_RESET_TIMEOUT_DAYS: '********'
    PREPEND_WWW: False
    ROOT_URLCONF: 'testdjango.urls'
    SECRET_KEY: '********'
    SECURE_BROWSER_XSS_FILTER: False
    SECURE_CONTENT_TYPE_NOSNIFF: False
    SECURE_HSTS_INCLUDE_SUBDOMAINS: False
    SECURE_HSTS_PRELOAD: False
    SECURE_HSTS_SECONDS: 0
    SECURE_PROXY_SSL_HEADER: None
    SECURE_REDIRECT_EXEMPT: []
    SECURE_SSL_HOST: None
    SECURE_SSL_REDIRECT: False
    SERVER_EMAIL: 'root@localhost'
    SESSION_CACHE_ALIAS: 'default'
    SESSION_COOKIE_AGE: 1209600
    SESSION_COOKIE_DOMAIN: None
    SESSION_COOKIE_HTTPONLY: True
    SESSION_COOKIE_NAME: 'sessionid'
    SESSION_COOKIE_PATH: '/'
    SESSION_COOKIE_SECURE: False
    SESSION_ENGINE: 'django.contrib.sessions.backends.db'
    SESSION_EXPIRE_AT_BROWSER_CLOSE: False
    SESSION_FILE_PATH: None
    SESSION_SAVE_EVERY_REQUEST: False
    SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'
    SETTINGS_MODULE: 'testdjango.settings'
    SHORT_DATETIME_FORMAT: 'm/d/Y P'
    SHORT_DATE_FORMAT: 'm/d/Y'
    SIGNING_BACKEND: 'django.core.signing.TimestampSigner'
    SILENCED_SYSTEM_CHECKS: []
    STATICFILES_DIRS: []
    STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',
     'django.contrib.staticfiles.finders.AppDirectoriesFinder']
    STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'
    STATIC_ROOT: '../staticroot'
    STATIC_URL: '/static/'
    TEMPLATES: [{'APP_DIRS': True,
      'BACKEND': 'django_jinja.backend.Jinja2',
      'DIRS': ['/home/srhaque/dev/testdjango/templates'],
      'OPTIONS': {'app_dirname': 'templates',
                  'auto_reload': True,
                  'autoescape': True,
                  'bytecode_cache': {'backend': 'django_jinja.cache.BytecodeCache',
                                     'enabled': False,
                                     'name': 'default'},
                  'constants': {},
                  'extensions': ['jinja2.ext.do',
                                 'jinja2.ext.loopcontrols',
                                 'jinja2.ext.with_',
                                 'jinja2.ext.i18n',
                                 'jinja2.ext.autoescape',
                                 'django_jinja.builtins.extensions.CsrfExtension',
                                 'django_jinja.builtins.extensions.CacheExtension',
                                 'django_jinja.builtins.extensions.TimezoneExtension',
                                 'django_jinja.builtins.extensions.UrlsExtension',
                                 'django_jinja.builtins.extensions.StaticFilesExtension',
                                 'django_jinja.builtins.extensions.DjangoFiltersExtension',
                                 'jinja_extensions.DebugExtension'],
                  'filters': {},
                  'globals': {},
                  'match_extension': ('.html',
                                      '.jinja'),
                  'match_regex': '^(?!admin/).*',
                  'newstyle_gettext': True,
                  'tests': {},
                  'translation_engine': 'django.utils.translation',
                  'undefined': None}},
     {'APP_DIRS': True,
      'BACKEND': 'django.template.backends.django.DjangoTemplates',
      'DIRS': ['/home/srhaque/dev/testdjango/templates'],
      'OPTIONS': {'context_processors': ['django.template.context_processors.debug',
                                         'django.template.context_processors.request',
                                         'django.contrib.auth.context_processors.auth',
                                         'django.contrib.messages.context_processors.messages']}}]
    TEST_NON_SERIALIZED_APPS: []
    TEST_RUNNER: 'django.test.runner.DiscoverRunner'
    THOUSAND_SEPARATOR: ','
    TIME_FORMAT: 'P'
    TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
    TIME_ZONE: 'UTC'
    USE_ETAGS: False
    USE_I18N: True
    USE_L10N: True
    USE_THOUSAND_SEPARATOR: False
    USE_TZ: True
    USE_X_FORWARDED_HOST: False
    USE_X_FORWARDED_PORT: False
    WSGI_APPLICATION: 'testdjango.wsgi.application'
    X_FRAME_OPTIONS: 'SAMEORIGIN'
    YEAR_MONTH_FORMAT: 'F Y'
    is_overridden: <bound method Settings.is_overridden of <Settings "testdjango.settings">>
    include: 
        ('myapp.tasks', 'celery.app.builtins')

ShaheedHaque commented 6 years ago

I tried to debug this, and noticed that the shell seems to think that the configured beat schedule is empty:

$ celery -A myapp shell
...
>>> celery.conf.beat_schedule
{}

Could it be that the "inspect scheduled" command only considers (statically) configured schedules?

ShaheedHaque commented 6 years ago

OK, this is weird. I start beat like this:

celery -A paiyroll beat --scheduler django_celery_beat.schedulers:DatabaseScheduler -l info --detach

(I've omitted --pidfile, --logfile and --workdir), and yet, the shell shows this:

$ celery -A myapp shell
>>> pprint.pprint({k:v for k,v in celery.conf.items() if k.startswith('beat')})
{'beat_max_loop_interval': 0,
 'beat_schedule': {},
 'beat_schedule_filename': 'celerybeat-schedule',
 'beat_scheduler': 'celery.beat:PersistentScheduler',
 'beat_sync_every': 0}

Now, I think the system is working because when I submit scheduled tasks (using the django CrontabSchedule and PeriodicTask model APIs), they show up in the admin and run as expected. Is the problem that the running copy of beat takes its cue from the command line, whereas everything else is using the static configuration?

ShaheedHaque commented 6 years ago

I changed the static configuration to look like this, but 'inspect scheduled' still shows nothing:

$ celery -A myapp shell
>>> import pprint;pprint.pprint({k:v for k,v in celery.conf.items() if k.startswith('beat')})
{'beat_max_loop_interval': 0,
 'beat_schedule': {},
 'beat_schedule_filename': 'django',
 'beat_scheduler': 'django_celery_beat.schedulers:DatabaseScheduler',
 'beat_sync_every': 0}
>>> 
now exiting InteractiveConsole...
$ celery -A myapp inspect scheduled
-> celery@baba: OK
    - empty -

ShaheedHaque commented 6 years ago

Light dawns...as per StackOverflow:

The celery inspect module appears to only be aware of the tasks from the workers perspective

So what's needed is a new command, possibly 'inspect periodic' which queries Beat for the ~scheduled~ periodic tasks. Is that right? If so, the relevant patch would be to the Celery project, not Beat, right?

jheld commented 6 years ago

Without knowing much... I would assume the following:

beat will have to expose its current schedule
flower will then need to look at the beat "workers" when reading these for the web app.

ShaheedHaque commented 6 years ago

Yes. To expand on the Beat part, obviously the nominal schedule is in the "external" database, so the request here is about querying the beat process' "internal" database for the actual state of the system.

I think the Flower part is separate, needed for the flower issue specifically.

chescales commented 5 years ago

It would actually be great to see the periodic tasks. Have you found any other way to see them so far?

ShaheedHaque commented 5 years ago

No. That was a primary reason for ditching my use of beat.

otniel commented 5 years ago

Any updates on this?

jheld commented 5 years ago

I'd be willing to give it a shot, but won't have anything ready before end of next week. If anyone is willing to, please go for it!

On Fri, Jul 12, 2019, 5:57 PM otniel notifications@github.com wrote:

Any updates on this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/celery/django-celery-beat/issues/126?email_source=notifications&email_token=AAGKLDPLNHRQELX2HUPBDBLP7D45BA5CNFSM4EQEGFL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ27F4Y#issuecomment-511046387, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGKLDLYFJ6K35LHLAMXJQTP7D45BANCNFSM4EQEGFLQ .

yshaban-godaddy commented 4 years ago

+1 for this feature. We cannot see scheduled tasks in things like flower. This is obnoxious as we are unable to see results in flower and are forced to assign results to our database to see the results.

jheld commented 4 years ago

Part of celery 5's design goals is to enable this. Unsure if/when, but it's up for improvement.

auvipy commented 4 years ago

Part of celery 5's design goals is to enable this. Unsure if/when, but it's up for improvement.

we might have to introduce some celery 5 stuff in celery 4.x branch which will not disrupt the API's much.

jheld commented 4 years ago

For this portion (excluding those who do the debugging to understand what pieces need to change in the current design to make it work), do we have any partially illuminating docs? I'd imagine this isn't super challenging. We would need to ensure the pidbox system is running, and write logic to "convert" the beat pending task "queue" so that it will respond to the active & queue etc commands, at the very least.

vumaasha commented 4 years ago

No. That was a primary reason for ditching my use of beat.

What alternative are you using currently?

ShaheedHaque commented 4 years ago

On Sat, 13 Jun 2020, 07:34 vumaasha, notifications@github.com wrote:

No. That was a primary reason for ditching my use of beat.

What alternative are you using currently?

I use systemd's timer units. That is in preference to cron, on essentially the same basis as discussed here, which is that systemd has a cleaner management story.

—

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/celery/django-celery-beat/issues/126#issuecomment-643579029, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABFHWKPB4DNIXC2AAU3Y4GLRWMMYDANCNFSM4EQEGFLQ .

Dhruv97Sharma commented 1 year ago

I tried to debug this, and noticed that the shell seems to think that the configured beat schedule is empty:
$ celery -A myapp shell
...
>>> celery.conf.beat_schedule
{}
Could it be that the "inspect scheduled" command only considers (statically) configured schedules?

Thanks! This worked for me!

celery / django-celery-beat

"celery -A myapp inspect scheduled" does not list periodic tasks #126