As a GCNotify developer,
I want to tweak Celery configuration
So that I get improved performance around processing and sending notifications.
WHY are we building?
There are reports from Veteran Affairs that propose a few Celery tweaks to improve the processing of the notifications. This is matched with a previous pull request opened by one of our past developers but never merged since.
WHAT are we building?
Small tweak in how the Celery queues are managed and polled. The current implementation is SQS, so we need to be mindful of these. It could be interesting to explore a different queue implementation as well but this is outside the scope of this task.
VALUE created by our solution
Faster processing of our notifications.
Acceptance Criteria
[ ] Processing of notifications is faster.
[ ] The reliability and stability of the processing pipeline get no visible impact.
Kyle MacMillan (VA Notify Tech Lead):
USA VA Notify made some changes! Celery (SQS) efficiency has been abysmal. We've had to scale out to keep pace with our growth, which is great, but we have seen Celery Task CPU usage averaging 10%. The culprit has been queues. We're going to revisit workers and queues further, but reducing the number of empty queues is something I would recommend all Notify forks investigate. Of particular note in the PR is the QueueNames and Config class changes to Celery settings.
Decreased the number of tasks needed by over 80% with no degraded performance.
Description
As a GCNotify developer, I want to tweak Celery configuration So that I get improved performance around processing and sending notifications.
WHY are we building?
There are reports from Veteran Affairs that propose a few Celery tweaks to improve the processing of the notifications. This is matched with a previous pull request opened by one of our past developers but never merged since.
WHAT are we building?
Small tweak in how the Celery queues are managed and polled. The current implementation is SQS, so we need to be mindful of these. It could be interesting to explore a different queue implementation as well but this is outside the scope of this task.
VALUE created by our solution
Faster processing of our notifications.
Acceptance Criteria
QA Steps
Additional Info
Additional investigation
As an additional investigation, we could check if the number of queues could be related to the performance drop that we experienced from this reported issue #178 https://app.zenhub.com/workspaces/notify-planning-core-6411dfb7c95fb80014e0cab0/issues/gh/cds-snc/notification-planning-core/178