SC #11 Monitoring and Maintenance

[jgreben]

What do we need to do to make sure Sinopia keeps running after the feature development cycles come to an end?

• [x] Determine SLA level (https://consul.stanford.edu/pages/viewpage.action?title=Digital+Library+Services+Catalog&spaceKey=DLSC) and fill out SLA entry for Sinopia
• [ ] Check to ensure that DevOps docs are complete (see: https://github.com/sul-dlss/DevOpsDocs/tree/sinopia/projects/sinopia), https://github.com/LD4P/sinopia/issues/236
  • [ ] Add a diagram of all the sinopia repo pieces, deployment containers, and docs on what talks to what how
  • [ ] Add hints on debugging problems across diff AWS containers
• [x] Communicate our SLA level and policies to wider community and get feeback. UPDATE: began communication today. We need to clarify Operations vs Engineering response commitments, and also determine how users can notify Operations about system outages and notify Engineering about work stoppage items; and also explain what we consider a work stoppage item
• [x] if possible redirect help.sinopia.io to github help wiki site so that if site is down, people can still use the shortcut URL to know how to get help LD4P/sinopia_editor#1843
• [x] CloudWatch: describe current monitoring and what is missing/needed
• [ ] Ability to backups and restore Trellis, including documentation on how to do it.
  • [ ] Document that most of the metadata lives in Postgres and is restorable that way
  • [ ] Document restoring Memento backup
• [ ] Ability to backups and restore Cognito, including documentation on how to do it.
• [ ] Determine training that is needed by developers and/or Ops to support Sinopia at desired SLA level.
  • ensure whole team is capable of doing "the stuff" for sinopia (deployments, setting up local laptops to do what is nec)
• [x] Create deployment instructions for dev, stage, and prod to include all sinopia dependencies (e.g editor, trellis, indexing pipeline, exporter, profile editor, etc.)

[kamchan]

• RDS (sinopia-production-rds) is being backed up. Daily snapshots done at 3AM and is kept for up to five days.

• trellis EFS is being backed up nightly at 10PM. They are kept within the AWS Backup (trellis-production-efs) resource for up to 97 days before expiring.

• Cognito users are also backed up to an S3 bucket (sinopia-cognito-production) and backed up every 8 hours. Manually looking at the older S3 bucket backups shows users information etc.

• elasticsearch backups are automatically done hourly for version 5.3 or above. We are on version 6.5.

The following are cloudwatch monitoring that are configured to send alerts to both sinopia-dev@lists.stanford.edu and dlss-dev-alerts@lists.stanford.edu

sinopia-homepage-production-elb-response-slow sinopia-homepage-production-cpu-high-alert sinopia-homepage-production-cpu-high sinopia-homepage-production-elb-health-hosts sinopia-pe-production-elf-health-hosts sinopia-pe-production-cpu-high sinopia-pe-production-elb-response-slow sinopia-pe-production-cpu-high-alert trellis-production-elb-health-hosts trellis-production-mem-high-alert trellis-production-elb-response-slow

[justinlittman]

@kamchan Do we we monitor that the ElasticSearch cluster is green?

[justinlittman]

We should add Honeybadger.

[ndushay]

• really clear deployment instructions (dev, stage, prod)
• was there a thing that ops was doing to help us that confused dev/stage/prod? is this something we can address with documentation (if I'm even remember correctly)
• diagram of all the sinopia repo pieces, deployment containers, and docs on what talks to what how

[ndushay]

• ensure whole team is capable of doing "the stuff" for sinopia (deployments, setting up local laptops to do what is nec)
• hints on debugging problems across diff AWS containers

[kamchan]

@justinlittman The elasticsearch cluster being yellow is because there is a single node in that cluster. There is a list of monitoring (ie cpu, java mem usage, kibana search checks) that will allow us to monitor that service.

[jgreben]

• ensure whole team is capable of doing "the stuff" for sinopia (deployments, setting up local laptops to do what is nec)
• hints on debugging problems across diff AWS containers

@ndushay I added some of your points to be included in the checklist at the top.

[kamchan]

Following elasticsearch monitoring checks have been added to prod/stag/dev.

sinopia-es-production-healthy-nodes-alert sinopia-es-production-jvm-mem-alert sinopia-es-production-cpu-high-alert

[michelleif]

@jgreben is this epic complete?

[michelleif]

@jermnelson can we close this?

[jermnelson]

@michelleif - closing this issue.