Create alarms for performance/stability metrics

[rkorytkowski]

We need to have a number of alarms setup for all envs:

• [x] Slow running DB queries > 1s
• [x] Memory utilisation for containers > 80%
• [x] CPU utilisation for containers > 90%
• [ ] Long running tasks > 12h
• [ ] Long pending tasks > 1h

[rkorytkowski]

@snyaggarwal I assume we don't log anything to identify long pending or running tasks?

Could celery beat easily run a task to log warnings e.g. WARN task with ID is pending for 1h and WARN task with ID is running for 12h?

[snyaggarwal]

Yes by inspecting workers we should be able to write something.

[paynejd]

@rkorytkowski is this done?