How to respond to a hosting emergency

[alanshaw]

A TABLEFLIP managed hosting site is unresponsive. What can you do?

Step 0 - Prerequisites

• Public key is in the ansible users' authorized_keys file
• Admin login to Digital Ocean
• Access to secrets.yml for the deployed app

  Step 1 - Inform

• Do not panic. You will successfully resolve the issue. It takes as long as it takes.

• Email the client to tell them a problem has been identified

  Dear CLIENT,

  We've identified an issue with your website. We're investigating the problem now and will keep you up to date with progress.

  Step 2 - Diagnose

• Check status.digitalocean.com to see if digital ocean is down
• Sanity check, is it still available for you?
  • See if you can ping the machine ping domain.com
  • See if you can send HTTP traffic curl domain.com
  • downforeveryoneorjustme.com
• Log onto the server ssh ansible@domain.com
• Is the service running?
  • Check status service SERVICE_NAME status
• What's in the logs?
  • Save the log if you see an error! It'll be useful later.
  • Get last 1000 lines sudo tail -1000 /var/log/upstart/SERVICE_NAME.log
  • Follow the logs as they happen sudo tail -f /var/log/upstart/SERVICE_NAME.log

    Step 3 - Resolve

• Service not running?
  • Follow the logs as they happen (see above)
  • Try start the service sudo service SERVICE_NAME start
• Machine completely unresponsive?
  • Login to Digital Ocean and restart the machine using Power > Power Cycle
• Machine still completely unresponsive?
  • Login to Digital Ocean and take a snapshot Snapshots > Take Snapshot
  • Restore from backup Backups > Restore from Backup
• Machine still completely unresponsive?
  • Create new VM in Digital Ocean
  • Use infrastructure project to deploy
  • Update DNS for site to point at new box
• Email the client to inform them of the diagnosis and resolution