search

cds-snc / notification-planning-core

Project planning for GC Notify Core Team
0 stars 0 forks source link

Create 15 Minute Soak Test for Staging #105

Open ben851 opened 1 year ago

ben851 commented 1 year ago

Description

As a developer/operator of Notify, I would like to ensure that my changes merged to staging do not have any impact to the system.

WHY are we building?

When deploying terraform changes, there can be a significant delay before the changes take effect. A simple smoke test may not identify issues as it runs very quickly.

WHAT are we building?

Create a new test scenario where the system is run under a sustained load for a period of time (15 minutes?) We may be able to leverage the existing performance test scripts running at reduced load. Integrate the soak test job into the notification-terraform merge to staging.

VALUE created by our solution

Increased stability of GC Notify.

Acceptance Criteria

QA Steps

ben851 commented 1 year ago

Ben to QA today in staging during K8s upgrade

sastels commented 1 year ago

PR https://github.com/cds-snc/notification-api/pull/1877

ben851 commented 1 year ago

Verified this is working when upgrading staging to 1.24 k8s. I will just add a bit of documentation on setup.

sastels commented 1 year ago

I'll make a "soak test" note in LastPass with good params