department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

280 stars 195 forks source link

Create Backup and Recovery Plan #475

Open ricetj opened 5 years ago

ricetj commented 5 years ago

If the whole system comes crashing down, how do we:

Recover
Restore backups
etc.

The goal of the issue is slightly vague, but we can refine as we look into this.

ricetj commented 5 years ago

Comment by ShawnArnwine Sunday Jul 24, 2016 at 18:37 GMT

Mostly covered in our ATO-related SOPs and IRP/ISCP...but needs to be turned into checklists and properly assigned.

ricetj commented 5 years ago

Comment by dkhenry Monday Jan 23, 2017 at 22:56 GMT

We should also test out the recovery

Things to Backup/Recover

[ ] Infrastructure Generation
[ ] Pet Instances ( CSR's )
[ ] RDS
[ ] Utility Instance Generation
[ ] Dev/Prod/Staging Instance Generation
[ ] S3 Deployment

ricetj commented 5 years ago

(09/5/2017)

[ ] Set up Matt and Wyatt with access to the test AWS govcloud accounts
[ ] Document existing procedures and capabilities. Send to Matt and Wyatt for review

ricetj commented 5 years ago

(11/29/2017) Pet instances (CSR's) basically just need to be snapshotted so we can restore them if needed.

ricetj commented 5 years ago

(01/17/2018) blocked by DRP tabletop, will result in more specific tix to support this effort

ricetj commented 5 years ago

(05/16/2018) Still relevant?

I lean toward yes, but will leave this up to Wyatt.

I'm envisioning a brief breakdown of the types of state and systems we manage, and how we generally provide backups/restoration for those systems. Critical persistent state in RDS and whichever EBS volumes we rely on, and how we restore infrastructure with terraform/ansible. It would be useful to summarize what we'll restore from scratch vs what we need to back up.

ricetj commented 5 years ago

@wyattwalter Ported this ticket over. This looks like more of an ATO issue, but may come into the ops scope. Wanted to get your take on this. Would this be a shared effort with Ops and Product Support?

andreahewitt-odd commented 4 years ago

@ricetj Does this ticket need VSP work?

ricetj commented 4 years ago

@andreahewitt-odd This is another ticket that was ported over from Vets.gov repo. https://github.com/department-of-veterans-affairs/vets.gov-team/issues/1542 This will most likely be an ATO ticket in the future but may touch a few teams. I think we need to wait till we have ATO up and going before taking this on.

ricetj commented 4 years ago

@ricetj to put this on top of the list of things to do.

jhouse-solvd commented 3 years ago

@ricetj Is this still active / pending / to-be-discovered?

ricetj commented 3 years ago

This is something that needs to be in the Q yes. Something the team needs to at the very least has an eye on for the future.

rmtolmach commented 1 year ago

We should create new initiatives to address this based on the state of current infrastructure. Leaving this until we can create new issues for that work.