Build a virtual appliance for Dispatch

Introduces an ova make target, which builds an OVA for dispatch-sever using the scripts under scripts/ova.

The OVA has two disks. The first is treated as the root filesystem and the second is used for application config files, data, and logs.

The OVA is Photon-based, and includes a variety of dependencies, the dispatch-server binary, and additional utility binaries: toolbox (a lightweight VMware guest tools implementation), ovfenv (for accessing OVF environment properties), and rpctool (for managing guestinfo).

Systemd units handle configuration of storage and networking.

Basic configuration information, such as the initial root password, is passed to the OVA via environment properties.

The OVA is built using a container. The build.sh script can be called from the container (e.g., during CI) with the ova-ci argument or will launch the container itself with the ova-dev argument.

The OVA build process creates raw disk images, installs the base OS, and then configures the OS in application-specific ways. The disk images are then converted into VMDKs. Next, placeholder information is replaced in an OVF file. Lastly, the files are compressed into an OVA.

Fixes: #654, #667, #669, #670

Testing

make ova output

$ make ova
GOOS=linux go build -ldflags "-X github.com/vmware/dispatch/pkg/version.version=v0.1.24-12-gf69ac31a -X github.com/vmware/dispatch/pkg/version.buildDate=2018-10-12T16:13:27Z -X github.com/vmware/dispatch/pkg/version.commit=f69ac31ad752619e1531d2c0351053b17369568a" -o bin/dispatch-server-linux ./cmd/dispatch-server
GOOS=linux go build -ldflags "-X github.com/vmware/dispatch/pkg/version.version=v0.1.24-12-gf69ac31a -X github.com/vmware/dispatch/pkg/version.buildDate=2018-10-12T16:13:27Z -X github.com/vmware/dispatch/pkg/version.commit=f69ac31ad752619e1531d2c0351053b17369568a -X github.com/vmware/dispatch/pkg/dispatchcli/cmd.imagesB64=H4sIAJcqwVsAA42RSxKDIBBE95yCC2g+7limsskmuxwAZVREBwMaK7cP+fg3ljuK6Z5X060kCkZP3MKl4AkQ5AUwilpAZr3QfROhIwXmZnJGhbQlr6I0Nk7VaKN2AyHb+3v/cCQ5x6R2q9otxPM8ohYx5bNKNQYbOEPlFxQMQL/pGkk3YGwKeb4FNhYv8DrBCjLjD74B1snmmPdoAJgXRMKWeZ3Vtuxrk5oYR0WYGitZwBlKQAEYSbCMUFpwlDHYilED99o97D9Gn84UM2lh2f85e+Lsw3wBpVEiqLECAAA=" -o bin/toolbox ./cmd/toolbox
GOOS=linux go build -ldflags "-X github.com/vmware/dispatch/pkg/version.version=v0.1.24-12-gf69ac31a -X github.com/vmware/dispatch/pkg/version.buildDate=2018-10-12T16:13:27Z -X github.com/vmware/dispatch/pkg/version.commit=f69ac31ad752619e1531d2c0351053b17369568a -X github.com/vmware/dispatch/pkg/dispatchcli/cmd.imagesB64=H4sIAJcqwVsAA42RSxKDIBBE95yCC2g+7limsskmuxwAZVREBwMaK7cP+fg3ljuK6Z5X060kCkZP3MKl4AkQ5AUwilpAZr3QfROhIwXmZnJGhbQlr6I0Nk7VaKN2AyHb+3v/cCQ5x6R2q9otxPM8ohYx5bNKNQYbOEPlFxQMQL/pGkk3YGwKeb4FNhYv8DrBCjLjD74B1snmmPdoAJgXRMKWeZ3Vtuxrk5oYR0WYGitZwBlKQAEYSbCMUFpwlDHYilED99o97D9Gn84UM2lh2f85e+Lsw3wBpVEiqLECAAA=" -o bin/rpctool ./cmd/rpctool
GOOS=linux go build -ldflags "-X github.com/vmware/dispatch/pkg/version.version=v0.1.24-12-gf69ac31a -X github.com/vmware/dispatch/pkg/version.buildDate=2018-10-12T16:13:27Z -X github.com/vmware/dispatch/pkg/version.commit=f69ac31ad752619e1531d2c0351053b17369568a -X github.com/vmware/dispatch/pkg/dispatchcli/cmd.imagesB64=H4sIAJcqwVsAA42RSxKDIBBE95yCC2g+7limsskmuxwAZVREBwMaK7cP+fg3ljuK6Z5X060kCkZP3MKl4AkQ5AUwilpAZr3QfROhIwXmZnJGhbQlr6I0Nk7VaKN2AyHb+3v/cCQ5x6R2q9otxPM8ohYx5bNKNQYbOEPlFxQMQL/pGkk3YGwKeb4FNhYv8DrBCjLjD74B1snmmPdoAJgXRMKWeZ3Vtuxrk5oYR0WYGitZwBlKQAEYSbCMUFpwlDHYilED99o97D9Gn84UM2lh2f85e+Lsw3wBpVEiqLECAAA=" -o bin/ovfenv ./cmd/ovfenv
./scripts/ova/build.sh ova-dev
--------------------------------------------------
starting docker dev build container...
2018-10-12 23:13:35 [=] Starting appliance build.
2018-10-12 23:13:36 [=] create disk images
2018-10-12 23:13:36  [==] creating dispatch-disk1.img
2018-10-12 23:13:36   [===] allocating raw image of 6GiB
2018-10-12 23:13:36   [===] wiping existing filesystems
2018-10-12 23:13:39   [===] creating bios boot partition
2018-10-12 23:13:40   [===] creating linux partition
2018-10-12 23:13:41   [===] reloading loop devices
2018-10-12 23:13:41   [===] formatting linux partition
mke2fs 1.42.13 (17-May-2015)
2018-10-12 23:13:42   [===] mounting partition /dev/loop5p2 at /tmp/tmp.kkZW1ucPXB//mnt/root
2018-10-12 23:13:42   [===] setup grup on boot disk
2018-10-12 23:13:42   [===] install grub to /tmp/tmp.kkZW1ucPXB//mnt/root/boot on /dev/loop5
Installing for i386-pc platform.
Installation finished. No error reported.
2018-10-12 23:13:46   [===] configure grub
2018-10-12 23:13:46  [==] creating dispatch-disk2.img
2018-10-12 23:13:46   [===] allocating raw image of 2GiB
2018-10-12 23:13:46   [===] wiping existing filesystems
2018-10-12 23:13:48   [===] creating linux partition
2018-10-12 23:13:49   [===] reloading loop devices
2018-10-12 23:13:49   [===] formatting linux partition
mke2fs 1.42.13 (17-May-2015)
2018-10-12 23:13:50   [===] mounting partition /dev/loop8p1 at /tmp/tmp.kkZW1ucPXB//mnt/data
2018-10-12 23:13:50 [=] Installing base os
2018-10-12 23:13:50  [==] building base
2018-10-12 23:13:50  [==] install OS to /tmp/tmp.kkZW1ucPXB/mnt/root
2018-10-12 23:13:50  [==] preparing install stage
2018-10-12 23:13:50   [===] configuring tdnf
2018-10-12 23:13:50   [===] initializing rpm db
2018-10-12 23:13:50   [===] configuring yum repos
2018-10-12 23:13:50   [===] verifying yum and tdnf setup
2018-10-12 23:14:01   [===] installing filesystem bash shadow coreutils findutils
/var/tmp/rpm-tmp.nCX5vk: line 3: cp: command not found
2018-10-12 23:14:26   [===] installing systemd linux-esx tdnf ca-certificates sed gzip tar
2018-10-12 23:15:17   [===] installing tzdata glibc-lang vim
2018-10-12 23:15:31   [===] installing system dependencies
Failed to connect to bus: No such file or directory
groupadd: group 'wheel' already exists
2018-10-12 23:16:22   [===] installing package dependencies
2018-10-12 23:16:52   [===] installing root
2018-10-12 23:16:52  [==] exporting base
2018-10-12 23:16:52 [=] Installing application layer
2018-10-12 23:16:52  [==] building app
2018-10-12 23:16:52  [==] run in chroot build-app.sh
2018-10-12 23:16:52  [==] setting mountpoints and adding build scripts
2018-10-12 23:16:52   [===] copying provisioners
2018-10-12 23:16:52  [==] configuring os
2018-10-12 23:16:52   [===] importing local gpg key
2018-10-12 23:16:52   [===] setting umask to 022
2018-10-12 23:16:52   [===] setting root password
2018-10-12 23:16:52   [===] configuring password expiration
2018-10-12 23:16:52   [===] configuring UTC timezone
2018-10-12 23:16:52   [===] configuring en_US.UTF-8 locale
Generating locales...
  en_US.ISO-8859-1... done
  en_US.UTF-8... done
Generation complete.
2018-10-12 23:16:53   [===] configuring haveged
2018-10-12 23:16:53   [===] configuring sshd
2018-10-12 23:16:53  [==] running provisioners
2018-10-12 23:16:53   [===] running script-provisioners/0-package_provisioning.sh
installing pyyaml
Collecting pyyaml
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
Installing collected packages: pyyaml
  Running setup.py install for pyyaml: started
    Running setup.py install for pyyaml: finished with status 'done'
Successfully installed pyyaml-3.13
You are using pip version 8.1.2, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
installing - docker compose
######################################################################## 100.0%
installing - jq
######################################################################## 100.0%
2018-10-12 23:17:06   [===] running script-provisioners/1-filesystem_provisioning.sh
2018-10-12 23:17:06   [===] running script-provisioners/2-provision_govc.sh
######################################################################## 100.0%
govc_linux_amd64.gz: OK
2018-10-12 23:17:12   [===] running script-provisioners/3-system_settings.sh
Created symlink from /etc/systemd/system/dispatch-appliance.target.wants/toolbox.service to /usr/lib/systemd/system/toolbox.service.
Created symlink from /etc/systemd/system/dispatch-appliance-ready.target.wants/dispatch-mounts.target to /usr/lib/systemd/system/dispatch-mounts.target.
Created symlink from /etc/systemd/system/local-fs.target.wants/dispatch-mounts.target to /usr/lib/systemd/system/dispatch-mounts.target.
Created symlink from /etc/systemd/system/dispatch-mounts.target.requires/repartition.service to /usr/lib/systemd/system/repartition.service.
Created symlink from /etc/systemd/system/dispatch-mounts.target.requires/resizefs.service to /usr/lib/systemd/system/resizefs.service.
Created symlink from /etc/systemd/system/dispatch-appliance-ready.target.wants/dispatch-appliance-environment.service to /usr/lib/systemd/system/dispatch-appliance-environment.service.
Created symlink from /etc/systemd/system/dispatch-appliance.target.wants/dispatch-appliance-ready.target to /usr/lib/systemd/system/dispatch-appliance-ready.target.
Created symlink from /etc/systemd/system/dispatch-appliance.target.wants/dispatch-appliance-tls.service to /usr/lib/systemd/system/dispatch-appliance-tls.service.
Created symlink from /etc/systemd/system/network-online.target.wants/sshd_permitrootlogin.service to /usr/lib/systemd/system/sshd_permitrootlogin.service.
Created symlink from /etc/systemd/system/dispatch-appliance-ready.target.wants/ovf-network.service to /usr/lib/systemd/system/ovf-network.service.
Created symlink from /etc/systemd/system/network-online.target.wants/ova-firewall.service to /usr/lib/systemd/system/ova-firewall.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
Created symlink from /etc/systemd/system/dispatch-appliance.target.wants/dispatch-server.service to /usr/lib/systemd/system/dispatch-server.service.
Created symlink from /etc/systemd/system/default.target to /usr/lib/systemd/system/dispatch-appliance.target.
Refreshing metadata for: 'VMware Photon Linux 1.0(x86_64)'
Refreshing metadata for: 'VMware Lightwave 1.0(x86_64)'
Refreshing metadata for: 'VMware Photon Linux 1.0(x86_64)'
Refreshing metadata for: 'VMware Photon Extras 1.0(x86_64)'
Cleaning repos: photon-extras photon-updates lightwave photon
Cleaning up everything
2018-10-12 23:17:22  [==] cleaning up base os disk
Cleaning repos: photon-extras photon-updates lightwave photon
Cleaning up everything
Initializing machine ID from random generator.
2018-10-12 23:17:32   [===] cleaning up tmp
2018-10-12 23:17:33   [===] removing man pages
2018-10-12 23:17:33   [===] removing any docs
2018-10-12 23:17:33   [===] removing caches
2018-10-12 23:17:33   [===] removing bash history
2018-10-12 23:17:33   [===] cleaning log files
2018-10-12 23:17:33   [===] clearing last login information
2018-10-12 23:17:33   [===] resetting bashrs
2018-10-12 23:17:33   [===] resetting ssh host keys
2018-10-12 23:17:33   [===] zero out free space
dd exit code 1 is suppressed
2018-10-12 23:17:33   [===] syncing fs
2018-10-12 23:17:33  [==] cleanup installer
2018-10-12 23:17:33   [===] remove build scripts
2018-10-12 23:17:33 [=] export images to VMDKs
2018-10-12 23:17:33  [==] exporting dispatch-disk1.img to dispatch-disk1.vmdk
2018-10-12 23:17:33   [===] unmount /tmp/tmp.kkZW1ucPXB//mnt/root
2018-10-12 23:17:33   [===] release loopback device /dev/loop5
2018-10-12 23:17:33   [===] converting raw image dispatch-disk1.img into dispatch-disk1.vmdk
2018-10-12 23:18:18  [==] exporting dispatch-disk2.img to dispatch-disk2.vmdk
2018-10-12 23:18:18   [===] unmount /tmp/tmp.kkZW1ucPXB//mnt/data
2018-10-12 23:18:18   [===] release loopback device /dev/loop8
2018-10-12 23:18:18   [===] converting raw image dispatch-disk2.img into dispatch-disk2.vmdk
2018-10-12 23:18:19  [==] VMDK Sizes
2018-10-12 23:18:19  [==] 354M  ./dispatch-disk1.vmdk
152K    ./dispatch-disk2.vmdk
2018-10-12 23:18:19 [=] --------------------------------------------------
2018-10-12 23:18:19 [=] packaging OVA...
2018-10-12 23:18:19  [==] updating version number
2018-10-12 23:18:19  [==] updating image sizes
2018-10-12 23:18:19  [==] rebuilding OVF manifest
2018-10-12 23:18:22 [=] build complete
2018-10-12 23:18:24  [==] SHA256: ea28d720e40d14ef9196019237b1bca1a0f550f98332199705ed39f39d1342d5
2018-10-12 23:18:25  [==] SHA1: a82abea60bf5d27bd963918b2ce98019ac088e67
2018-10-12 23:18:26  [==] MD5: eec886c1c8603da7cb6347828845a7a5
2018-10-12 23:18:26  [==] 354.648MB
2018-10-12 23:18:26 [=] --------------------------------------------------
2018-10-12 23:18:26 [=] cleaning up...
2018-10-12 23:18:26 [=] removing dev loops and images

ovftool deploy output

$ ovftool --datastore=sharedVmfs-1 --noSSLVerify --acceptAllEulas --name=dispatch-v0.1.24-12-gf69ac31a --diskMode=thin --powerOn --X:waitForIp --X:injectOvfEnv --X:enableHiddenProperties --prop:appliance.root_pwd=ova-test-root-pwd --prop:appliance.permit_root_login=True --net:"Network"="VM Network" scripts/ova/bin/dispatch-v0.1.24-12-gf69ac31a.ova "vi://administrator@vsphere.local@10.192.207.166/dc1/host/cluster1"
Opening OVA source: scripts/ova/bin/dispatch-v0.1.24-12-gf69ac31a.ova
The manifest validates
Enter login information for target vi://10.192.207.166/
Username: administrator%40vsphere.local
Password: ********
Opening VI target: vi://administrator%40vsphere.local@10.192.207.166:443/dc1/host/cluster1
Deploying to VI: vi://administrator%40vsphere.local@10.192.207.166:443/dc1/host/cluster1
Transfer Completed                    
Powering on VM: dispatch-v0.1.24-12-gf69ac31a
Task Completed                        
Received IP address: 10.192.204.37
Completed successfully

manual testing

$ cat ~/.dispatch/config.json
{
    "current": "dev",
    "contexts": {
        "dev": {
            "host": "10.192.204.37",
            "port": 8443,
            "scheme": "https",
            "organization": "dispatch",
            "cookie": "",
            "insecure": true,
            "namespace": "dispatch"
        }
   }
}
$ ./bin/dispatch-linux create base-image nodejs dispatchframework/nodejs-base:0.0.9 --language nodejs
Created base image: nodejs
$ ./bin/dispatch-linux create image nodejs-empty nodejs
Created image: nodejs-empty
$ ./bin/dispatch-linux create function nodejs-func examples/nodejs/hello.js --image nodejs-empty
Created function: nodejs-func
$ ./bin/dispatch-linux get functions
     NAME     |                       FUNCTIONIMAGE                       | STATUS |         CREATED DATE          
--------------------------------------------------------------------------------------------------------------
  nodejs-func | dispatch/func-6906995f-c462-4e59-bbb5-a39671c149d0:latest | READY  | Fri Oct 12 16:53:38 PDT 2018  
$ ./bin/dispatch-linux exec nodejs-func --wait
{
    "blocking": true,
    "executedTime": 1539388435,
    "faasId": "6906995f-c462-4e59-bbb5-a39671c149d0",
    "finishedTime": 1539388435,
    "functionId": "47110463-359c-4a19-b698-ce0b98773268",
    "functionName": "nodejs-func",
    "input": {},
    "logs": {
        "stderr": null,
        "stdout": null
    },
    "name": "4f1e4088-935c-4a68-9575-a1a6d1fda769",
    "output": {
        "myField": "Hello, Noone from Nowhere"
    },
    "reason": null,
    "secrets": [],
    "services": null,
    "status": "READY",
    "tags": []
}

Notes

I put the new code under scripts as discussed, but wonder if ova should actually be a sibling directory instead of a child.
There are certainly some unnecessary dependencies currently being installed. The current version erred on the side of over-inclusion, to avoid any potential annoying-to-debug issues due to missing dependencies. We should clean these up.
Because most of the build process was copied from VIC, there are probably a few lingering references to VIC. We should clean these up.
The current version does not include a custom "DCUI". It should be possible to build one (again, based on VIC's implementation), so that users get some helpful information when the look at the guest console. However, that should involve some design around what information would be the most useful to display.
I preserved the VIC EULA. I'm not sure that's needed.
Currently, dispatch runs and stores data at /var/lib/dispatch. The VIC OVA was designed with separate storage for logs, data, and databases under /storage/. I've preserved the VIC design here as I think having those things separate has some advantages (e.g., logs can't fill up a partition and cause the database to get into a bad state). We should think about how best to tailor this to Dispatch.

vmware-archive / dispatch

Build a virtual appliance for Dispatch #680

Testing

Notes