🔥 Remove Celery and Flower, they are currently not used nor recommended

[tiangolo]

🔥 Remove Celery and Flower, they are currently not used nor recommended

When choosing Celery I evaluated RQ, Dramatiq, and a couple of other options, but Celery was the only one at the time that would allow having workers running externally with potentially different dependencies (e.g. TensorFlow models), as there was a way to make Celery not import everything for the client sending a job.

Nevertheless, that is not the default or documented behavior in Celery. And Celery also has many features that I don't use, or would not suggest using, like chains and chords, as they became quite difficult to debug and handle when I used that.

I would prefer something different, but I don't know yet what would be the best alternative. I wanted to try ARQ, as it's from the same author of Pydantic, but that's also a smaller library without any type of dashboard/observability, etc.

I want to try Prefect, Hatchet, and maybe others (Dask, Ray, Dagster?). I still don't know what is the best option.

For things that are not resource-intensive and are expected to be done very fast, a simple integrated BackgroundTask from FastAPI would be enough. For heavier problems that require distributed tasks running in the background, I think there might be alternatives better than Celery. I have to investigate the subject more before deciding and implementing anything here.

For now, I just don't want to suggest using something that I currently don't endorse, and that is also not really needed in this repo as it is. :sweat_smile:

[PierrickLP]

🔥 Remove Celery and Flower, they are currently not used nor recommended

Why are they not recommended ?

[tiangolo]

@PierrickLP I was just writing all the descriptions above. :sweat_smile:

Once I figure out what would be the approach I would recommend, I would update it in this repo. :nerd_face:

[Haiz14]

There's a celery file not deleted https://github.com/tiangolo/full-stack-fastapi-template/blob/master/backend/app/celeryworker_pre_start.py

[copdips]

maybe it's time to build something called fastflow

[flying-sheep]

Maybe check this out: https://github.com/taskiq-python/taskiq

I don’t know if it fits the bill, but it at least is built with async support. It has quite some plugins that make it support all important queues, as well as things like FastAPI integration:

[PhilippWu]

@tiangolo do you have new insights about Task Queueing and the tooling you will add for that? I am building up ab project witch needs queing and delaying and prioitising of queues with workers. Is there any desiccion with tool you may use in the future?

[copdips]

If a beautiful UI matters, both Airflow and Prefect are excellent choices.

[Ccolina03]

@tiangolo Do you have any recommendations yet? :)

[qiangxinglin]

@tiangolo Bump, any news?

[CIRDL]

@tiangolo Also very curious if you have any updates 😁