search

s3rius / FastAPI-template

Feature rich robust FastAPI template.
MIT License
1.9k stars 164 forks source link

Add a way to send tasks to Redis (celery ?) #106

Closed sorasful closed 1 year ago

sorasful commented 2 years ago

Hi there !

I find it very nice to have Redis/RabbitMQ but it seems that for now we are a route to get and set values in these services, but we don't really have a way to send background tasks to them easily.

Do you think it would be a good idea/possible to add a way to send and consume background tasks easily ? I was thinking about Celery but maybe there are others options that could be worth.

What do you think ?

s3rius commented 2 years ago

Hi. You have several ways of doing this. Have you tried default fastapi background tasks? https://fastapi.tiangolo.com/tutorial/background-tasks/

Celery and Dramatiq are nice , but these libraries don't support async functions (You can overpass this restriction using async_to_sync function from asgiref) .

Often I write workers by hand. by creating python file that listens to specific queue.

sorasful commented 2 years ago

Hey! Yeah I already use fastapi background tasks because as you said, it supports async and it's already included. That's very cool. But I think that it has some limitations, meaning you may need to spawn multiple instances of your FastAPI app to handle these tasks to avoid slowing down your API too much.

Also, if for some reason your app crashes or reboots, you loose all the tasks which is not ideal.

I heard about dramatiq, never tried it though. I thought Celery was supposed to support async, but it seems to have been delayed. I'll see what I can find.

s3rius commented 2 years ago

Hi there. After some research I found out that there is no such thing as celery or dramatiq for async projects. So I've created one. It's called taskiq. Currently it's unstable, but I will try to add this integration after we stabilize all interfaces, add tests, and create more brokers and result backends.

It took only month to build it from scratch. But I guess it may be very helpful project for many people that struggle in search for the async celery alternative.

haffi96 commented 1 year ago

Whats the benefit/differences of taskiq compared to rq? https://python-rq.org/ or perhaps https://github.com/samuelcolvin/arq if you want built in async

s3rius commented 1 year ago

@haffi96 RQ supports only redis. Also taskiq can pipeline tasks using taskiq-pipelines library and it's easily extendable.

haffi96 commented 1 year ago

It looks great. Will definitely consider it next time i have a need. Side note, thoughts on adding hacktoberfest tag to the repo? (definitely asking for my own selfish desire to get a badge... 👀 )

s3rius commented 1 year ago

@haffi96 Added a tag. Thanks for noticing.

s3rius commented 1 year ago

Hello, guys. It's been a long ride, but finally taskiq is mature enough to be integrated in fastapi-template.

To close this issue I've created a project, lol, but now I'll be trying to make it more popular and continue developing it. There're some missing features in taskiq, but everything what we plan for future will be implemented.

It doesn't have a docs about how to use it with fastapi in official documentation, please use this README as reference to integrate it in existing projects.

Warm thanks for your patience and have a great hacking.