Postmortem: 7/25, 7/26, and 7/30 Prod Offline Incidents

gracekretschmer-metrostar commented 3 months ago

User Story or Problem Statement

The VA needs to understand the root cause of each prod outage and the steps the CMS team is taking to prevent these outages in the future.

What is Known

CMS prod went offline on 7/25, 7/26, and 7/30 CMS began the process to undergo the certificate renewal on 7/25 Jenkins went offline on the evening of 7/25 and was brought back online on 7/26 Certificate renewal was finished on 7/29

Relevant Links

Cert Renewal steps: https://vfs.atlassian.net/wiki/spaces/~712020436a1be296514af5b58f0846a4d9d910/pages/3327295549/Procedure+to+Renew+cms.va.gov+Cert+on+Staging+and+Prod
Cert Renewal Github ticket: https://github.com/department-of-veterans-affairs/devops/pull/14579
Jenkins platform support ticket: https://dsva.slack.com/archives/CBU0KDSB1/p1722009476229769
7/25 prod offline slack thread: https://dsva.slack.com/archives/CT4GZBM8F/p1721937609972519
7/26 prod offline slack thread: https://dsva.slack.com/archives/CT4GZBM8F/p1722008142269569
7/30 prod offline slack thread: https://dsva.slack.com/archives/CT4GZBM8F/p1722369813108219

Steps for Implementation

Research the relevant links and data dog to determine source of each outage.
Document findings in postmortem report.

Acceptance Criteria

[x] The root cause of why prod CMS went offline on 7/25, 7/26, and 7/30.
[x] A solution is implemented that resolves prod CMS from going offline during daily deploy.
[x] Opportunities to proactively prevent prod CMS from going offline in the future.
[x] Postmortem report submitted to Github repo.

gracekretschmer-metrostar commented 3 months ago

Confluence page for the draft postmortem: https://vfs.atlassian.net/wiki/spaces/PCMS/pages/3353247840/2024-07-25+25+30+CMS+Prod+Offline+Incidents

gracekretschmer-metrostar commented 3 months ago

Easyretro board: https://easyretro.io/publicboard/NqWKfJWUr2dFRoCJjX2ybXcRyws2/0e6b627e-5e1d-4583-bafd-e12193cedd88

7hunderbird commented 1 month ago

I have a PR ready to review for this https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/pull/1969

department-of-veterans-affairs / va.gov-cms