celery-beat, DatabaseScheduler and per-tenant schedule

[Guest007]

Hi! We make easy solution for tenants and schedules in database (like periodic tasks).

In case of:

1. django-tenants
2. tenant-schemas-celery
3. database-driven tasks scheduler (DatabaseScheduler)

you can't work in separate tenants with their own periodic tasks directly, without some dispatcher task or some separate model in public tenant, like django-tenants-celery-beat use.

With customized DatabaseScheduler you can use celery-beat individually for each tenant as it was usual in non-tenant environment. Just remember that django_celery_beat must be in your TENANT_APPS.

[maciej-gol]

you mean it is impossible to have periodic tasks stored in the database, on a per-tenant basis?

[Guest007]

You can, but only from public will work as expected

[maciej-gol]

Do you have the periodic tasks inserted dynamically into the DB, or are they sourced from your source code?

[Guest007]

It was tested with tasks, described in model PeriodicTask (part of django-celery-beat). In that way, task must be defined in source code, but conditions of use - in PeriodicTask. As I see, this is common scenario for manipulate task's schedule in terms of DatabaseSchedule (part of django-celery-beat) https://django-celery-beat.readthedocs.io/

[maciej-gol]

I believe the tenant_schemas_celery.scheduler.TenantAwareScheduler scheduler class does what you need. When you don't specify the tenant_schemas key, or set it to None, the task will be sent to all the tenants. Am I missing something?

[Guest007]

Yes. You missed django-celery-beat - Celery Periodic Tasks backed by the Django ORM (as wrote in About section). It's simple and clear. Except one - celery-beat don't know about tenants.

And if my clients = tenants - I have a way to make manipulation with task's schedule through admin individually by client.

One client want to run, for example, some data imports each day at 3, but other - only weekly. And third client don't use this task at all.

Project https://github.com/QuickRelease/django-tenants-celery-beat collects all tasks from all tenants to one model, placed in public and run tasks from that place. I guess it overhead.

celery-beat-tenants-scheduler just run through all tenants to work.

[maciej-gol]

Since you are adding the tasks dynamically to the DB, have you tried adding the `_schema_name: "" kwarg to the task?

[Guest007]

How can it helps me? How can it helps standard DatabasScheduler to go to tenants and find model with PeriodicTasks there? Your variant may be helpfull if PeriodicTasks with tasks for all tenants placed in public and we get schema for make context from kwargs. But this is not our variant of use

[maciej-gol]

The DatabaseScheduler pulls tasks from the public schema and sends them onto the queue, with given arguments. You can save the tasks schedule into the public schema with a proper _schema_name kwarg so that the scheduler will send the celery task with a proper schema inside.

[Guest007]

My customized DatabaseScheduler pulls tasks from all schemas, not only from public and send them onto queue. No need to save tasks from one schema to another, or make any other job. It just works. Your variant is simple and great. Customized DatabaseScheduler is just a tenant-aware replacement for standard DatabaseScheduler. In other cases your TenantAwareScheduler is very helpfull.

[Alihassanc5]

@Guest007 thank you for this code, could you please share your TENANT_APPS and SHARED_APPS settings? If i want to run the tasks in all tenants except public tenant how can I do that? because when I am trying to start celery beat it gives me migrations error (probabbly because I placed celery beat in TENANT_APPS only.

[Guest007]

@Guest007 thank you for this code, could you please share your TENANT_APPS and SHARED_APPS settings?

For example in settings.py

PROJECT_APPS = [
    "some_your_app",
    ...
]

INSTALLED_APPS = [
    "django.contrib.admin",
    ...
    "django_celery_beat",
    ....
] + PROJECT_APPS

TENANT_SPECIFIC_APPS = [
    "django_tenants",
    "customers_as_in_django_tenants_docs",
]

SHARED_APPS = TENANT_SPECIFIC_APPS + INSTALLED_APPS  # in public schema
TENANT_APPS = INSTALLED_APPS  # in tenant schemas for each client.
INSTALLED_APPS = SHARED_APPS

TENANT_MODEL = "customers_as_in_django_tenants_docs.Client"
TENANT_DOMAIN_MODEL = "customers_as_in_django_tenants_docs.Domain"

CELERY_BEAT_SCHEDULER = "celery_beat_tenants_scheduler.scheduler.TenantDatabaseScheduler"

If i want to run the tasks in all tenants except public tenant how can I do that? ...

You can put on top of task's body something like that:

from django.db import connection

def some_not_public_task():
    if not hasattr(connection, "schema_name") or connection.schema_name == get_public_schema_name():
        return
    # body of all_tenants_task

[Alihassanc5]

Thanks again, what's the prupose of adding django_celery_beat in both apps?

[Guest007]

Thanks again, what's the prupose of adding django_celery_beat in both apps?

1) Using all apps in public schema (SHARED_APPS) is the easiest way to make tests 2) The 'public' schema is the same schema as all the others schemas/tenants. Therefore the app is included everywhere. You can do it differently