jonrh
commented
8 years ago Completed as of c48e27c68781a5bfb2a7927fe68a49117756e531. Took quite a bit of work. Many issues on the way. For example that Docker containers run as root users as default and that doesn't jive well with Celery which uses insecure pickle. Had to learn about linking Docker containers.
The approach I went with was basically the ideal way, 3 separate containers. After some reading I discovered we could reuse the lovelace/backend Docker image for the Celery worker. We simply overwrite the execution command to fire up the Celery worker instead of running the Flask server.
Some comments on my work can be seen in the _test_builddeplay.sh file.
Xinqi has completed writing the Celery tasks. It's in the file backend/Celery/tasks.py. For the setup Xinqi choose to use Redis as a message broker (a queue to store unprocessed tasks).
What remains is to set it up on our production environment.
I see three possible ways to do it given that we have a Docker environment.
The ideal way: 3 separate containers. Celery task runner, Flask backend, Redis queue. The middle way: 2 containers. Flask backend + Celery task runner, Redis queue. The dirty way: 1 big container containing all.
There are pros and cons to the approaches. What I'm aiming for is a good mix of simplicity and flexibility. Originally I tried to go for the 3 container approach. However the cons with that plan is we would have to maintain a separate Dockerfile for the Celery task runner which would complicate the Jenkins build'n'deploy script. We'd also have to create a separate Docker Hub account because it only allows one per account (currently used up by lovelace/backend). The Celery task runner also shares a lot of the same Python dependencies as the backend so it might sense to have it contained in the same container.
I will now attempt the middle way because I think the "dirty" way could actually entail more work.
Links that might be of help: https://github.com/RichardBronosky/celery_redis_demo