Deploy new celery-sms-send pods in prod

[sastels]

Description

As a Notify team member, I need to change our prod sms configuration to be able to scale email and sms independantly.

WHY are we building?

Need to be able to scale emails and sms independantly.

WHAT are we building?

deploy the sms-send pods in prod

VALUE created by our solution

We will be able to keep both emails and sms close peak send rates close to but not over our AWS limits

Acceptance Criteria

• [x] staging load tests show 2000-3000 emails / min and 500-600 sms fragments / min
• [ ] celery-sms-send pods running on production with this configuration

QA Steps

• [x] verify) that there are no sms sent in the priority-tasks, bulk-tasks, nor send-sms-tasks queues (otherwise the 30 celery pods will be sending them!)
• [x] verify) that almost all sms are sent using the celery-sms-send pods
• [x] email send rate on prod between 2000 and 3000 emails / min
• [x] sms peak send rate on prod between 500 and 600 fragments / min

[sastels]

Various tests as documented here Main points:

• increasing concurrency increased resource usage but not send rate
• sms-send pods would scale up but then scale down in the middle of a send

We tweaked the pod requirements in the hope that that would keep them spun up. This seemed to work on 5 pods.

[sastels]

promising results on ten pods with tweaked hpa. spun up 10 pods and kept them up. Some strange bumps at the start and end that we might be able to smooth out by lowering the rate limit down to 1/s (current 5/s)

Private Zenhub Image

[sastels]

Test

• 20K emails
• 4K sms

Configuration

• 30 max celery pods
• 11 max sms-send pods
• 4 concurrency for everyone, 1/s rate limit for deliver_sms (as in prod now)
• 3 on demand nodes, rest spot instances

Results

• Karpenter ramped up to 7 nodes
• after ramp up system reached a steady 2900 emails / min, 570 sms / min

Private Zenhub Image

Private Zenhub Image

[sastels]

So I suggest that for V1 we ship the above configuration without karpenter and with a constant 7 nodes. After this we can

• test / tweak karpenter in staging (in particular the ramping down behaviour)
• scale up the email pods in prod (for 73 pods will need to go to 10 nodes as per karpenter tests)
• scale up sms pods in prod (and perhaps get some more long codes if that's possible)
• do some work to better understand how the rate limiting works and how we should configure the sms-send celery workers and pods

[sastels]

use sms send deployment on prod (well, after the next release) https://github.com/cds-snc/notification-manifests/pull/2041

[sastels]

merged to staging!

[sastels]

Test of 40K emails, 60K sms on staging Results as expected:

• system spun up to 30 celery pods, 11 celery-send-sms
• constant 8 nodes
• CPU peaked at 62%
• no stalled messages at the end (no pods restarted)
• climbed to expected send rates

Private Zenhub Image

Private Zenhub Image

• A Cloudwatch query) showed that the sms were sent via the send-sms-low queue as expected
• Another query) showed that the celery-sms-send pods were used to send the sms

[sastels]

A double 80K / 12K test got similar results.

[jimleroyer]

First iteration will go in production today with a sending capacity of around 3,300 for emails. Steve will focus on the deployment and test for today.

[sastels]

Release went pretty well. Two issues:

• email pods wouldn't scale down
• sms pods wouldn't scale all the way up (went 3->4 instead of 3->11)

Email issue fixed, sms to investigate...

[ben851]

Ben to QA

[ben851]

Verified that the sms sends are in the sms-send-high and medium queues. Verified that virtually all sms are being sent by the celery-sms-send pods (there are a few that aren't; these are notify admin 2fa) Verified over the last 3 days that peak email send rates were approx 2000 per minute Checked SMS Send over the past week, and it peaked around 300. I'm assuming this is due to lack of load. Will confirm w/ Steve at standup