VA.gov 2/11/21 Site Outage - Post Mortem - Githubissues

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

284 stars 206 forks source link

VA.gov 2/11/21 Site Outage - Post Mortem #19824

Open johnhashva opened 3 years ago

johnhashva commented 3 years ago

What Happened

While attempting a content update to the VA.gov website on Thursday 2/11/21 at approximately 7:15 a.m. ET, a technical issue with the CSS file caused the website to not render properly. The VA.gov technical team quickly responded to resolve the issue. The website is accessible again as of 9:00 a.m. ET. The team is monitoring the website status and will review and address issues that contributed to it.

What We Need to Know

Root cause
How the issue was discovered? https://dsva.slack.com/archives/CBU0KDSB1/p1613050241052700 Kevin Hoffman: At 8:30 am after a DM from Lucas Tickner, I alerted the on duty FE and DevOps leads in #vfs-platform-support.
How was the team alerted?
How speedy and effective was resolution?
What can be done to prevent from happening again?

Technical Details (TBD)

Process Details (TBD)

Recommended Action Steps (TBD)

annaswims commented 3 years ago

Only those with a pagerduty account can page the person on call or even find out who they are.

MickinSahni commented 3 years ago

Capturing a question from the on-call thread for posterity

How do "we trigger an incident via snow or other oit channel" i.e. how do we let others know we're working on the issue at and?

RachaelMR commented 3 years ago

Was it a Drupal content update? Or vets-website?

omgitsbillryan commented 3 years ago

~We were notified at 8:19am EST via PagerDuty (slack link).~

Correction: @annaswims manually triggered this alert, it was not an automated one

mchelen-gov commented 3 years ago

crosslinking https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/blob/master/Postmortems/2021-02-11-VA.gov%20Site%20Outage.md can everyone access that?

jhouse-solvd commented 3 years ago

Ticket to track exploration/implementation of advanced monitoring that could catch this (and similar) scenario:

Monitoring / Alerting - Check for broken links and assets on VA.gov (in response to recent outage) #19843