Build realistic load tests

[sastels]

Description

As a developer, I need to be able to run load tests in staging similar to normal production behaviour. So that we can be sure that our changes are tested in a production like scenario.

WHY are we building?

We want to catch more problems (in particular wrt infrastructure changes) before they happen in production.

We want to simulate production traffic to find issues that only appear after long-running production scenarios.

WHAT are we building?

Soak-style load tests that mimic production:

• occasional bulk jobs
• smaller normal sends
• high priority sends
• runs over a period of several hours
• Generates spikes in the system

VALUE created by our solution

Improved testing for releasing to production. More confidence in the prod release, fewer incidents related to change management.

Improved general stability.

Acceptance Criteria

• [x] Test sends occasional low priority email and sms jobs. Pods scaling should go up and down
• [x] Tests sends medium priority single and small bulk email / sms
• [x] Test sends high priority email / sms every few seconds
• [x] Test causes k8s nodes to scale up and down
• [x] Configurable run time

QA Steps

• [x] Run tests for 4 hours and analyze system behaviour.
• [x] Determine how accurately the test emulates production over an X hour period (To improve for the future)
• [ ] Run through the readme to verify it is accurate.

[sastels]

started roughing in some rough tests of occasional high priority sms / email or bulk low priority email / sms jobs

[sastels]

https://github.com/cds-snc/notification-api/pull/2032

[sastels]

still draft but almost ready for review

[sastels]

need to add medium notifications to this, then will move to review.

[sastels]

ready for review https://github.com/cds-snc/notification-api/pull/2032

[ben851]

Will review the PR today

[sastels]

in 1Password there's a file Notify Staging / Load Test Variables file

[ben851]

Ben to QA

[sastels]

Ben still to QA!