Feature: distributed task processing and dynamic module loading

[CodePint]

Added two new settings for defining dynamic task loading and switched to using signatures when building workflows. This will allow users to define custom task module locations and enable distributed task processing (api/worker code+infra can be separated)

1) DIRECTOR_CELERY_IMPORTS: list of arbitrary files/modules to load https://docs.celeryproject.org/en/stable/userguide/configuration.html#imports

2) DIRECTOR_CELERY_AUTO_DISCOVER: dynamic **/tasks module loading https://docs.celeryproject.org/en/stable/reference/celery.html#celery.Celery.autodiscover_tasks known bug (patched): https://github.com/celery/celery/pull/6424

refs: https://docs.celeryproject.org/en/stable/reference/celery.html#module-celery https://docs.celeryproject.org/en/stable/reference/celery.html#celery.signature

Signed-off-by: George Eddie geddie@linius.com

[ncrocfer]

Unfortunately same comment as https://github.com/ovh/celery-director/pull/134#issuecomment-1065361088

This feature adds a lot of complexities in director, this is not aligned with the primary goal of this project that is to be as simple as possible for new Celery users.

Can you explain us the needs behind this PR?

[anthonyolea]

I'm also confused a little bit. I'm not quite sure to understand the purpose of this PR for Celery Director when you have a DIRECTOR_HOME folder with a tasks folder inside. The tasks are already discovered by Celery Director. The idea of Celery Director was to declare how to use tasks by adding them in workflows into the workflows.yml.

Also, the Quickstart aimed to be a gently introduction to Celery Director, I don't think this kind of feature should be mentioned in this section.

[ncrocfer]

Thanks @CodePint but as discussed in the team and in this current discussion we take the decision to close this PR, we prefer to keep this project as simple as possible for new Celery users. This PR introduces too many complexities for them.