Scale up the delivery receipt processing

[jimleroyer]

Description

As a user of GCNotify, I want to know when my email or SMS were delivered in a timely manner, So that I get rapid feedback on the notification's delivery And react accordingly on success or failure.

WHY are we building?

The delivery receipts processing is not scaling up as much as what we can currently send, triggering potential traffic in our delivery receipt queue as the celery workers cannot process as fast and introducing delays in displaying the proper status of notifications or communicate callbacks to API users.

WHAT are we building?

Without dictating how we can do this, the what is about scaling up with a similar factor to the sending process or similar to incoming receipt volume, so that increased traffic does not increase delays with delivery receipts.

VALUE created by our solution

Better user experience on proper notification status and callbacks.

Acceptance Criteria

• [ ] An increased volume of delivery receipts should not take longer to process than the actual sending of the notifications.
• [ ] An increased volume of delivery receipts should not impede other normal operations handled by Celery workers such as the database saving operations (performed by Celery workers) or the actual sending of the notifications (performed by Celery workers).

QA Steps

• [ ] Perform Rollercoaster tests and monitor the processing of delivery receipts.

[sastels]

Testing raising the celery-scalable maxReplicas. Results here.

Summary:	test size	maxReplicas	peak wait time
50000	30	712
50000	40	601
50000	50	362
50000	60	300

It is likely that we'd get a better increase in performance by instead adding 30 pods dedicated just to processing delivery receipts. We can add a new deployment of pods, say celery-delivery-scalable that just processes the delivery receipts. (and leave the current celery-primary and celery-scalable pods as they are for now)