Outage caused by certificate change during SLA hours

[wunderhund]

Please read the Triage Rules of Engagement for instructions on what types of issue should be submitted using this template.

Status

UNRESOLVED

Severity Analysis

Estimate of how critical this bug is

• sev 2

User Impact

• Who is the user impacted by this problem?
  • All third-party developers who rely on dev-api.va.gov for testing and development of their products.
• How does this problem affect the user?
  • dev-api.va.gov became unresponsive and unavailable.

Description

• Change to the public certificate for dev-api.va.gov occurred during SLA hours (7am to 9pm Eastern Time), causing an outage for the site. This change was requested in INC6935855 by @b00klegger on August 7 and occurred on August 21 between 08/21/2019 05:01:59 PM ET to 08/21/2019 05:15:59 PM ET.

Proposed Solution

• Conduct a postmortem on this incident to determine why the dev environment's SLA hours were not communicated to the Gateway Operations team to prevent an outage.
• Determine what documentation and training is necessary to ensure all parties understand that the Dev environment has a strict SLA; no downtime of any kind is permitted between 7am-9pm ET.

Timeline

Background: the old lower environment certs for dev-api.va.gov, staging-api.gov, and so on will expire on August 30, 2019, so they needed to be replaced with new certificates.

July 15

• @wyattwalter opened issue va.gov-team/582 to organize the updating of SSL certs for various lower-environment certs

  July 17

• @tskinn submitted PR devops/5231, updating lower environment certs for the revproxy load balancers.

  August 2

• Ryan Watson submitted PR devops/5319 adding the certificate files to the devops repo.
• Taylor Skinner submitted PR devops/5321, changing the dev-api.va.gov certificate internally on the revproxy deployments.

  Aug 5

• The above PRs (5231, 5319, 5321) were all merged and applied to the internal environment (revproxy deployment and load balancers). No problems were reported.

  Aug 7

• Ryan Watson submits ServiceNow tickets requesting that the certs be updated on the public-facing side (TIC/F5). The tickets he submitted are:

  
  INC6935855

dev-devel INC6949035

dev INC6948888

staging-api INC6949284

staging-devel INC6949350

staging INC6949112

- David Young from GateWay Ops updated INC6935855 to say `currently awaiting the password from the requester in an encrypted format.`
##### Aug 8
- Ryan Watson supplied the new dev-api.va.gov cert and its password to the Gateway Ops team.
##### Aug 12
- David Young from Gateway Ops reports that he had pre-staged the new certificates for dev-api.va.gov on the F5 load balancers that serve the Gateway West and Gateway East TIC internet gateways.
##### August 20
- Frank Hopkins from Gateway Ops posts in the ticket to report that the maintenance is scheduled for 8/21/19 at 5pm ET. He also reports that a meeting invite was sent out, but it's not clear to whom.
##### August 21
- `5:20pm ET`: Zachary Jones from Gateway Ops reports in the ticket that the certificate has been updated successfully.
- `6:03pm ET`: Shawnee Petrosky sends a message to Craig Butler on the Lighthouse API Platform Infrastructure team reporting that a certificate changed caused an outage for the Health APIs from 08/21/2019 05:01:59 PM ET to 08/21/2019 05:15:59 PM ET. The error received for dev-api.va.gov was:```
Our Pingdom monitoring is alarming on an invalid certificate and running dev-api.va.gov through SSL Labs is noting that "This server's certificate chain is incomplete. Grade capped to B."```
- `6:39pm ET`: Craig Butler acknowledges this message and begins troubleshooting.
- `7:41pm ET`: Craig Butler identifies that the problem is likely a certificate change by the Gateway Ops team due to the upcoming expiration of the old dev-api.va.gov certificate.
- `8:07pm ET`: Craig Butler confirms to Shawnee Petrosky that the public-facing dev-api.va.gov certificate was changed by the Gateway Ops team during the period in question and that the ServiceNow ticket documenting the work is INC6935855.

[cvalarida]

Looks like this was an on-call incident, not a triage incident. (The naming is confusing, I admit.)

@ricetj Is this a thing for the operations team to take over?

[ricetj]

Looks like this was an on-call incident, not a triage incident. (The naming is confusing, I admit.)

@ricetj Is this a thing for the operations team to take over?

Hello @cvalarida we are short-handed right now, so I'm not sure If we could address this anytime soon. So on-call incidents that have unresolved issues are not run through triage?

[alexpappasoddball]

@omgitsbillryan Can you take a look at this and provide feedback