Add DO Droplet volumes

[katefike]

https://docs.digitalocean.com/products/volumes/getting-started/quickstart/

• Create a DO droplet volume for the DB

  /mnt/sage_db:/var/lib/postgresql/data

• Create a DO droplet volume for the MX

  /mnt/sage_mailserver/home/:/home/
  /mnt/sage_mailserver/var/log/:/var/log/

• Nuke the docker volumes on the droplet and ensure the data is still there on the:
  • db ✓
  • mx ✓
• Spin up a new droplet, mount the volumes, and ensure the data is still there on the:
  • db ✓
  • mx ✓ Followed this tutorial for moving volumes: https://docs.digitalocean.com/products/volumes/getting-started/quickstart/#move-volumes-to-different-droplets
    1. Turn off the droplet.
    2. Detach the two volumes from the original droplet.
    3. Detach the reserved IP from the original droplet.
    4. Take a final snapshot of the original droplet: prod-24041801-test-deletion
    5. Delete the droplet and the firewall.
    6. Create a new droplet. Run bash setup/2_create_prod_server.sh
    7. Note that two reserved IPs will be created. Unassign and delete the new one.
    8. Assign the existing reserved IP to the new droplet.
    9. Attach the volumes to the new droplet.
    10. Ensure the the local .env has the correct values for prod
    11. Run bash setup/3_configure_prod_server.sh
    12. Send a test email.

[katefike]

Related Create a docker volume for the mx container: https://github.com/katefike/sage/issues/28

[katefike]

DB ERRORS

sage-db          | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
sage-db          | It contains a lost+found directory, perhaps due to it being a mount point.
sage-db          | Using a mount point directly as the data directory is not recommended.
sage-db          | Create a subdirectory under the mount point.

Resolved by mounting a subdir of the mount dir.

/mnt/sage_db/data:/var/lib/postgresql/data

[katefike]

MX ERRORS

sage-mailserver  | /postfix_dovecot_config.sh: line 8: ![[: command not found
sage-mailserver  | find: '/etc/opendkim/domainkeys': No such file or directory
sage-mailserver  | useradd: user 'kfike' already exists
sage-mailserver  | New password: Retype new password: passwd: password updated successfully
sage-mailserver  |  * Reloading Postfix configuration...
sage-mailserver  |    ...fail!
sage-mailserver  |  * Restarting IMAP/POP3 mail server dovecot
sage-mailserver  |    ...done.
sage-mailserver  | mkdir: cannot create directory '/home/kfike/Maildir': File exists
sage-mailserver  | /usr/lib/python3/dist-packages/supervisor/options.py:470: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
sage-mailserver  |   self.warnings.warn(
sage-mailserver  | Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
sage-mailserver  | For help, use /usr/bin/supervisord -h
sage-mailserver exited with code 2
sage-mailserver  | /postfix_dovecot_config.sh: line 8: ![[: command not found
sage-mailserver  | find: '/etc/opendkim/domainkeys': No such file or directory
sage-mailserver  | useradd: user 'kfike' already exists
sage-mailserver  | New password: Retype new password: passwd: password updated successfully
sage-mailserver  |  * Reloading Postfix configuration...
sage-mailserver  |    ...fail!
sage-mailserver  |  * Restarting IMAP/POP3 mail server dovecot
sage-mailserver  |    ...done.
sage-mailserver  | mkdir: cannot create directory '/home/kfike/Maildir': File exists
sage-mailserver  | /usr/lib/python3/dist-packages/supervisor/options.py:470: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
sage-mailserver  |   self.warnings.warn(
sage-mailserver  | Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
sage-mailserver  | For help, use /usr/bin/supervisord -h
sage-mailserver exited with code 2

• /var/log/ in production branch in container

  kfike@prod:~/sage$ docker exec -it sage-mailserver bash
  root@15b5739b13d1:/# ls /var/log/supervisor/
  supervisord.log

• /mnt/sage_mailserver/var/log/ in this feature branch on host and in container

  
  kfike@prod:~/sage$ sudo ls /mnt/sage_mailserver/var/log/
  mail.log

- The problem is that the container mounts /mnt/sage_mailserver/var/log/, which lacks a supervisord.log file. Then the container crashes when this file isn't found.
  - is mail.log created on startup and supervisord.log isn't? -> yes
  - create a supervisord.log file at the same time as mail.log file in the entrypoint.sh script
  - what is supervisord.log even used for?

[katefike]

MX/TLS ERRORS

• Everything in Docker was nuked and then TLS broke. Some certs might be missing

  sage-mailserver  | Apr 18 22:11:37 prod postfix/smtpd[302]: warning: cannot get RSA certificate from file "/etc/letsencrypt/live/prod.sagefinancial.dev/fullchain.pem": disabling TLS support
  sage-mailserver  | Apr 18 22:11:37 prod postfix/smtpd[302]: warning: TLS library problem: error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:288:fopen('/etc/letsencrypt/live/prod.sagefinancial.dev/fullchain.pem','r'):
  sage-mailserver  | Apr 18 22:11:37 prod postfix/smtpd[302]: warning: TLS library problem: error:20074002:BIO routines:file_ctrl:system lib:../crypto/bio/bss_file.c:290:
  sage-mailserver  | Apr 18 22:11:37 prod postfix/smtpd[302]: warning: TLS library problem: error:140DC002:SSL routines:use_certificate_chain_file:system lib:../ssl/ssl_rsa.c:615:

• Indeed, no certs are present. Are they not being mounted to the container?

  (.venv) kfike@prod:~/sage$ docker exec -it sage-mailserver bash
  root@b6d872ecc26f:/# ls /etc/letsencrypt/live/prod.sagefinancial.dev/
  root@b6d872ecc26f:/# 

[katefike]

Old firewall

[katefike]

Instructions from DO about mounting the volume

Create a mount point for your volume mkdir -p /mnt/sage_db mkdir -p /mnt/sage_mailserver

Mount your volume at the newly-created mount point: sudo mount -o discard,defaults,noatime /dev/disk/by-id/scsi-0DO_Volume_sage-db /mnt/sage_db mount -o discard,defaults,noatime /dev/disk/by-id/scsi-0DO_Volume_sage-mailserver /mnt/sage_mailserver

Change fstab so the volume will be mounted after a reboot echo '/dev/disk/by-id/scsi-0DO_Volume_sage-db /mnt/sage_db ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab echo '/dev/disk/by-id/scsi-0DO_Volume_sage-mailserver /mnt/sage_mailserver ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

[katefike]

Sanity Check: is the volume mounted?

• Display the disk space allocated to the volume. If it's not mounted, there won't be an output: df -h /mnt/sage_db ✓
• Show all currently mounted filesystems: 'mount | grep '/mnt/sage_db' ✓