Shifting from Celery to RabbitMQ

Cloud-CV / CloudCV-Old

CloudCV - Large-Scale Distributed Computer Vision As A Cloud Service

http://cloudcv.org

MIT License

51 stars 37 forks source link

Shifting from Celery to RabbitMQ #35

Closed deshraj closed 7 years ago

deshraj commented 8 years ago

Since we want to load the model in another process and then do prediction so as the model don't get loaded every time a request has been sent from the user. Celery loads the model in the main process and do prediction in its sub-process(i.e the worker) and this leads to a race condition since the process is not fork-safe. So, we are shifting to RabbitMQ and it works well with that.

For more details, see this https://github.com/celery/celery/issues/3402

gauthamzz commented 7 years ago

@deshraj is this open ? can i try if it is ?

deshraj commented 7 years ago

@gauthamzz we are planning to rebuild the CloudCV website. So, we don't want to solve this issue now. We will let you once we have everything planned. I am glad to see that you are interested in solving the difficult issues. :-)