Scale celery based on SQS rather than cpu / memory

[sastels]

Description

As a develorer, I need the system to scale up and down reliably

WHY are we building?

Have the system scale up and down as need in a more predictable way

WHAT are we building?

conversation with Max:

Random musing - I am assuming your are scaling in and out based on CPU/Memory pressure? It would be interesting if you could scale based on SQS metrics instead. Presumably you would need to be able to ingress metrics from CloudWatch into k8s and then use those as policies to scale.

GKE has something like this https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/README.md

or - https://keda.sh/docs/2.11/scalers/aws-sqs/

VALUE created by our solution

better scaling then using cpu or memory metrics

Acceptance Criteria

• [ ] celery pods (celery and celery-sms-send) scale using SQS metrics rather than cpu or memory
• [ ] system scales up and down as expected during a load test

QA Steps

• [ ] under no load system is scaled down
• [ ] during a load test celery and celery-sms-pods scale up