Setup async job handling using Celery

[batpad]

Celery is the recommended and supported way to do asynchronous job processing as well as regularly occurring "jobs" on the backend. So far, we have used regular Linux cron-jobs to handle running commands at some periodicity to handle recurring daily / hourly jobs.

This has a number of issues:

• Runs on same machine as API, and can cause spiky resource strains on the machine when jobs run.
• Does not have a well-defined queueing and retrying system, logging is implemented ad-hoc and not built into the tool.
• Does not land itself to async tasks that need to be triggered by user action - eg. having strings translated when fields in the db are updated.

Celery provides a fairly easy to use framework to setup async tasks and periodic jobs. It will allow us to significantly better manage work queues, separate worker processes to a separate machine, and streamline logging, error handling and retry logic across tasks.

This is required for the Translation implementation as the async translation workflow cannot be run with the current Cronjobs since it needs to respond to user action. When we implement this for Translation, it will be fairly trivial to move the current cron-jobs over, and will allow us to manage all async tasks in a cohesive way, and set us up much better to be able to scale and manage tasks in the future.

A rough plan here:

• @thenav56 works to setup Celery in the code and locally, integrates with translation workflow
• @GregoryHorvath works with IT to provision a separate machine to run worker processes to separate it out from the API machine, and integrate with Azure hosted Redis for the job queue.

Once we have the above done and working, we can create a separate ticket to move the existing cron-jobs to the new Celery setup, which should hopefully be a fairly straightforward lift.

@thenav56 please let me know if this seems like an accurate description of the plan based on our discussion, and apologies for the delay in getting this ticket up.

[thenav56]

Update: The celery setup part is done through https://github.com/IFRCGo/go-api/pull/1359. And we can create a separate instance for celery on container infrastructure deployment.