search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 203 forks source link

Audit devops repo for un-versioned credstash lookups #12726

Closed dginther closed 3 years ago

dginther commented 4 years ago

Description

What details are necessary for understanding the specific work or request tracked by this issue? There was an outage caused by an un-versioned credstash lookup, when vets-api deployed, it pulled the latest evss prod key, which had been placed in credstash in preparation for the new key to be deployed. Since the new key had a higher version number than the old key, the new deploy pulled the new key, even though it was not ready to be deployed yet.

Background/context/resources

Any additional context for the reader to know, if applicable https://dsva.slack.com/archives/C30LCU8S3/p1598042685002000

Technical notes

Notes around work that is happening, if applicable

Tasks

Definition of Done

Reminders

ricetj commented 4 years ago

Need AWs access before taking this one.

omgitsbillryan commented 3 years ago

Might an equal & opposite outage occur if a lookup was versioned? In such a scenario, the developer updates the value in credstash but doesn't bother to change the version in the lookup. They expect the new version to be used, but instead get the old one.

Both scenarios hinder on the fact that the developer made an assumption about how the lookup occurred rather than specifically verifying it.

"I updated the value in credstash, therefore I assume that..."

I'm not sure one assumption is any more valid than the other. FWIW, I would tend to assume the latter 🤷.

jbritt1 commented 3 years ago

Removed cms related items from changes due to cms-infra team member request at this time (ref: https://dsva.slack.com/archives/CJYRZK2HH/p1609781236065600).

jbritt1 commented 3 years ago

Can we close this since we are moving to parameter store now? Not sure how much more time we want to invest in updating credstash versions vs changing to parameter store calls which is arguably out of scope and another ticket.

dginther commented 3 years ago

Closing, moving to Parameter store.