Move email delivery into their own dedicated queues

[sastels]

Description

As a Notify developer, I need email delivery tasks to be in their own queues so that I can treat them differently than other tasks (ex database, callbacks).

WHY are we building?

Moving email delivery to their own queues will allow us to scale up email delivery without scaling up other tasks such as database operations or callback processing. This will also allow better monitoring and alarming for email delivery issues.

WHAT are we building?

Move email delivery tasks to their own dedicated celery queues. Note that this task does not include moving the email delivery tasks to their own dedicated pool of celery workers.

VALUE created by our solution

We will be able to scale up email delivery without scaling up other parts of the system.

Acceptance Criteria

• [x] deliver_email() tasks go through their own celery queues.
• [x] wait time alarms for the new queues created
• [x] charts monitoring the new queues added to the email dashboard

QA Steps

• [x] verify that smoke tests still pass
• [x] Verify that the new queues are being used with a cloudwatch query such as this one)

[sastels]

PR to use the new queues in celery: https://github.com/cds-snc/notification-api/pull/1997

[sastels]

PR to add alarms for the new queues https://github.com/cds-snc/notification-terraform/pull/962

[sastels]

PR to add the queues to the email dashboard https://github.com/cds-snc/notification-terraform/pull/963

[sastels]

These PRs can wait to be merged / released after I'm back (Oct 23)

[ben851]

Jimmy approved and improved the PRs!

To be merged today after am release

[sastels]

in production

[jimleroyer]

Looks like it's working 👍

Private Zenhub Image