Change base image to Ubuntu, add Docker volume

[craig-willis]

Problem(s)

• CoreOS/Container Linux is EOL
• We frequently run out of local disk due to Docker image sizes
• The proposed solution for implementing MATLAB support will increase the image cache requirements and requires a more recent Docker version.

Fixes #44.

Approach

Modify the terraform deployment to use Ubuntu

• Change the base image from CoreOS to Ubuntu 20.04 LTS
• Port configuration from config.ign to pre-setup.sh
• Add external volume for Docker image cache

How to test

1. Provision new cluster using terraform_deployment (i.e, configure variables.tf and terraform apply)
2. Restore database from production backup
3. Confirm deployment end-to-end

[craig-willis]

Encountered a few problems while testing, recording for posterity.

• DNS error caused by bug in godaddy script. The script currently doesn't handle subdomains well, which was known at some point
• gwvolman expects Docker volumes to be in /var/lib/docker (see https://github.com/whole-tale/gwvolman/blob/master/gwvolman/utils.py#L242). We originally mounted in /mnt/docker, and this needs to be changed
• Ran into a known but old problem with gwvolman on Ubuntu

open: Permission denied
mount.davfs: can't mount using fuse kernel file system
mount.davfs: trying coda kernel file system
mount.davfs: no free coda device to mount
Traceback (most recent call last):
  File "/usr/local/bin/girderfs", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/girderfs/__main__.py", line 80, in main
    subprocess.check_output(cmd, shell=True)  # FIXME
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'echo "willis8
token:AaZdfVjQ2AnqsXYplrTnKm3IMh9FqJMjc3dh2Bv1XMTUm5FJABjUIPNKjbIqOiyK" | mount.davfs -o uid=1000,gid=100 https://girder.ubuntu.wholetale.org/tales/5cfd57fca18691e5d1feeda6 /mnt/docker/volumes/5cfd57fca18691e5d1feeda6_willis8_grDlaB/_data/workspace' returned non-zero exit status 1
[2020-08-21 17:59:11,680: INFO/ForkPoolWorker-6] Task gwvolman.tasks.create_volume[c0092072-2847-41b6-a7e4-96fd302fa6ed] succeeded in 4.102106210004422s: {'sessionId': '5f400b6c59d4748b20b28f53', 'mountPoint': '/mnt/docker/volumes/5cfd57fca18691e5d1feeda6_willis8_grDlaB/_data', 'volumeName': '5cfd57fca18691e5d1feeda6_willis8_grDlaB', 'nodeId': 'qt9c0dd7y3yc2b35q2e7xmqkr', 'instanceId': '5f400b6a59d4748b20b28f43'}

This had been fixed on deploy-dev (https://github.com/whole-tale/deploy-dev/commit/2d7ff65b) with --security-opt apparmor:unconfined.

• Ran into a problem with Docker and systemd-resolved on Ubuntu 18.04 and greater. Our solution is to:

add DNS and FallbackDNS to /etc/systemd/resolved.conf
sudo systemctl restart resolve-systemd.service
sudo rm /etc/resolv.conf
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

[craig-willis]

Thanks for catching and fixing that.