search

darwin-network / slash

1 stars 1 forks source link

upgrade to bionic #17

Closed slingamn closed 5 years ago

slingamn commented 6 years ago

The point release 18.04.1 is out; let's rebuild the server on Bionic. Proposed strategy:

  1. Get a new VM that runs Bionic
  2. Start a new branch of the repo, updating configs for Bionic compatibility
  3. Bring up services on the new host. E-mail is the trickiest. We should provision new DKIM keys for the new host (which will require new DKIM records in DNS). We probably don't need to worry about SPF before the changeover.
  4. Fully test the new server.
  5. Change the A record to point to the new server, wait 60 seconds, stop the ircd on the old server.
  6. Get certbot working on the new server.
  7. Close down the old server (we forgot about this for a couple months last time).
slingamn commented 6 years ago

Let's do #6 as part of this. Bionic ships openssl 1.1.0g, which includes CHACHA20 cipher suites. We can probably just use the "modern" cipher list from here: https://wiki.mozilla.org/Security/Server_Side_TLS

slingamn commented 5 years ago

@edmund-huber could you create these DNS records for testing purposes?

(1) an A record for bionic.darwin.network pointing to 167.114.129.198 (2) an AAAA record for bionic.darwin.network pointing to 2607:5300:201:3100:0:0:0:7427 (3) a TXT record for 201808._domainkey.darwin.network with contents:

k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDQ0j5RVXr6jtMjvPx72JTL/3p2CwqgqUK34m8WKzH+XxXLbSvlnWlziv6yVpE5IcAB8u48OXUzMAZxaBmhlHMPg8K5vadKyi7/8LQDRyE/RZvU5LCI6ztdvGj8ZUwdg8a16C/XOG1v6zo4wAtg3Le/prsJmHES+e99q1/A3y7IzwIDAQAB
slingamn commented 5 years ago

New server looks good. Steps for the cutover (maybe Monday afternoon):

  1. Warn users, communicate a fallback contact method
  2. Sync the ircd database over
  3. Reboot the new server one last time, check that everything is up
  4. Change over DNS (change the A record for darwin.network; create a new AAAA record for darwin.network)
  5. Wait at least 60 seconds
  6. Restart the ircd on the old server; this should force people to disconnect, then reconnect to the new server
  7. Verify that people are able to connect to the new server
  8. Set up certbot on the new server to do automated renewal; repoint /etc/certificates/irc.darwin.network to contain symlinks to /etc/letsencrypt/live
  9. Merge the bionic configs into master
  10. Leave the old server up as a hot spare for a bit, then come back and shut it down
slingamn commented 5 years ago

Preliminary checks look good. Here are the DNS changes (these should be as nearly simultaneous as possible, to avoid any split-brain):

  1. Change the existing A record for darwin.network to point to 167.114.129.198, keeping the TTL at 60 seconds
  2. Create a new AAAA record for darwin.network, pointing to 2607:5300:201:3100:0:0:0:7427, TTL 60 seconds
edmund-huber commented 5 years ago

Old darwin.network A record: 35.171.127.21. Old darwin.network AAAA record: N/A.

slingamn commented 5 years ago

Assigning to you to shut down the old server; if we have reliability problems with OVH, I think we'll spin up a new EC2 instance on Bionic rather than try to reuse the old one.

There are three entities involved:

  1. The EC2 instance
  2. The EBS volume
  3. The Elastic IP (this is subject to an hourly charge when not attached to a running instance)
slingamn commented 5 years ago

bump