Server Maintenance 2018-09-29

[cycomachead]

This is a set of notes / issues about the server maintenance on the weekend.

• [x] update lapis to latest + install rock spec
• [x] run all Ubuntu system updates
• [x] upgrade postgres to pg10
• [x] verify system hostname / setup process
• [x] script/steps to dump + restore DB
• [x] script to sync recently changed projects

[cycomachead]

Detailed(ish) Plan

• Finalize any changes to staging server
• [x] prepare coffee and tea
• [x] Start a timer
• [x] Set server to maintenance mode
• On existing snapcloud:
  • [x] as postgres: pg_dumpall --clean --if-exists > /2018-09-27.out
  • [x] as cloud: psql -d snapcloud -t -o projects_to_sync.txt -c "SELECT (id/1000) || '/' || id FROM projects WHERE lastupdated > now() - interval '5 days'" -- Adjust to 3 days, or simply do > 2018-09-27 00:00:00
• [x] Make a new droplet from a copy of the staging server
• [x] Bootup new snapcloud server
• [x] ensure the new server is the right size droplet (16GB 4 core)
• [x] Take a snapshot of the snapcloud droplet on DO
• On the current staging server:
  • [x] scp projects_to_sync.txt from the main server
  • [x] Run script to sync project files: There's a sync files script on the staging server
• On the new snapcloud server:
  • [x] Update hostname by following: https://askubuntu.com/a/87687
  • as cloud:
  • [x] Update .env for the proper config:
    • [x] update hostname
    • [x] update LAPIS_ENVRIONMENT
    • [x] update store path
  • [x] ensure current certs are in place.
• [x] restore data to pg instance on new server
  • [x] as postgres psql -f /2018-09-25-test-3-backup.out postgres
  • [x] use psql to verify counts of data transferred over
• [x] wait for project syncing to be done
• [x] unmount volume from snap-staging
• [x] resize project storage volume to 1TB in size (roughly 40% overhead for current use)
• [x] update nginx-amplify agent:
  • [x] hostname = cloud.snap.berkeley.edu in /etc/amplify-agent/agent.conf
  • [x] restart the agent (service amplify-agent restart)
• [ ] figure out letsencrypt
  • [ ] copy data to ~/lets-encrypt/snap-cloud.cs10.org
• [x] copy server logs to snapCloud/logs
• [x] start lapis server
• [x] test by visiting the IP address
• [x] update IP address in DO load balancer to new droplet
• [x] turn off old droplet

After a couple days

• [ ] delete the old droplet

[jmoenig]

Thanks, Michael! Do you think it's ready enough so we can notify users on the TEALS forum and on Piazza about the maintenance this upcoming weekend? If so I'd offer to do that, unless someone else wants to...

[cycomachead]

@jmoenig Yeah, this I think we should be able to do this. Could you please post the time as 11pm PST 9/28 to 4am PST 9/29? But I expect the actual migration to take only 2 hours.

I think starting a tiny bit earlier will be a little bit easier and better for me, in the event something tricky pops up.

[bromagosa]

Best of luck in the migration!

When I'm back to work you'll have to walk me through all these changes.

[cycomachead]

Of course! And thanks. :)