jimleroyer commented 11 months ago

Description

As an ops lead managing performance of GCNotify, I want to know why the system has a decreased sending rate compared to 18 months ago, Given the same infrastructure setup, So that I can understand what happened And fix the performance gaps if possible.

WHY are we building?

Increase the sending rate to what they were before, given the same infra setup.

WHAT are we building?

Investigating the decrease performance. We had an email sending rate of at least 1,800 emails / minute and recently, it was observed to be around 800 emails / minute.

VALUE created by our solution

Sending faster without increasing cost.

Acceptance Criteria

[ ] Investigate when the performance rate decreased.
[ ] Investigate potential source of performance decrease at these time periods.
[ ] Identify where the sending rate decreased (in our data pipeline? or at AWS sending stage?
[ ] If necessary, open AWS ticket to inquire on recent changes that might have affected our sending rate with AWS.
[ ] Fill out a section in the ADR on increasing sending rate with the investigation results.

QA Steps

[ ] Shared with the team and stamped with reviews approval.

jimleroyer commented 11 months ago

Jimmy to run send rates query in staging
Jimmy to provide visuals on long term trend of perf tests in staging

jimleroyer commented 10 months ago

The visuals were provided 10 days ago.
Jimmy to run local tests but has trouble running DD-API with API locally. Will meet with Steve today to troubleshoot.

ben851 commented 9 months ago

Jimmy is running tests on individual commits to try and identify when bottlenecks were created. Jimmy will upload test results to an s3 bucket for review.

ben851 commented 9 months ago

Running into virtual environment issues, need to downgrade the dependencies so that the old code will run correctly.

sastels commented 9 months ago

still testing, need to tweak the scripts to evaluate the tests.

jimleroyer commented 9 months ago

The tests are ran, have to extract the results in the db with proper timestamps. Started work on that and ongoing.

cds-snc / notification-planning-core