jimleroyer commented 3 months ago

Description

As an ops lead, I want to know what is the highest rate limit I can send SMS with GCNotify, So that I know how far we can push the system.

As a business owner, I need to know the current GCNotify SMS sending rate limit, So that I can adjust the daily and annual SMS limits.

WHY are we building?

We need to increase our SMS sending limit and for that, we need to know our latest capacity with the introduction of Pinpoint as a sending mechanism and the short code acquisition.

WHAT are we building?

We are testing as high as we can send of SMS via AWS pinpoint. Hence we might want to increase the number of Kubernetes pods and adjust our Karpenter/scale set configuration.

VALUE created by our solution

The ability for each service to send more SMS per year and on a daily manner.

Acceptance Criteria

[x] Come up with a testing strategy to test the limits within the GCNotify system.
[x] Come up with a testing strategy to test the limits with AWS.
[x] Come up with a testing strategy to have several real numbers.
[x] Test the highest limits that we can reach sending SMS.
[x] Identify the issue we hit at the SMS sending limit.
[x] Run the rollercoaster tests in the staging environment.
[ ] create a "monitor bottlenecks" dashboard
[ ] Potentially slowly and gradually increase the number of k9s pods in production with closed monitoring of selected metrics to make sure the system is stable, with rollback strategy if required.

QA Steps

[ ] Review the methodology used in the testing strategy,
[ ] Review the results afterward.

sastels commented 3 months ago

Testing first with internal test number. So going through Notify but NOT doing the boto call to Pinpoint. This will give us an idea of how much we can send before we hit Notify bottlenecks (database / network / k8s / ?).

Tested by uploading large (20K-40K) csvs of sms to 6135550123.

Data moved to this document

Summary: (note that the current state in production is 20 scalable pods)

primary pods	scalable pods	total pods	internal send rate	rate / pods
3	20	23	1250	54
3	30	33	1800	55
3	40	43	2320	54

sastels commented 3 months ago

staging remains set at max 40 scalable pods.

Future work:

We're still rate limiting sms to about one per second. Since we're getting about 54 sms / minute per pod, this rate limit appears to be per pod and not per worker? We should increase this rate limit and see what results we get. For example, say we keep 30 pods and triple the rate limit to 3/s. This setting is in the .env files, currently for both staging and production we have

CELERY_DELIVER_SMS_RATE_LIMIT=1/s

Also: I think we should throw sleep statements into celery 😱

The problem is that AWS has limits in fragments / second, not SMS / second.
We could add a
```
sleep ( (# fragments - 1) /  CELERY_DELIVER_SMS_RATE_LIMIT )
```
This would hopefully mean that the celery task took about the amount of time that the SMS would have taken if we sent its fragments one at a time. This way we wouldn't have to worry about large SMS causing AWS rate limit errors.