libvirt-bosh-cpi

A Go BOSH CPI for the libvirt virtualization API.

Motivation

Cloud Foundry is a very interesting tool and environment to target and use for development purposes. Cloud Foundry requires a BOSH CPI to deploy. Unfortunately, the existing BOSH CPI's seem to be development-only (that is, temporary), unsuitable for Cloud Foundry, or require a small cluster of machines, or deploy to a cloud provider.

Libvirt is a good compromise. The current intent is to keep it scoped at 1 host. If something larger exists, there are a number of solutions.

Features

• Libvirt settings are configurable. Copy manifests/libvirt-kvm-vars.yml or manifests/libvirt-qemu-vars.yml to setup custom configurations. Extend or alter as needed.
• Libvirt connectivity is configurable. Options include TCP or TLS connections. Unix sockets are available, but without being able to map the socket into the BOSH VM is unrealistic. See manifests/libvirt-socket.yml, manifests/libvirt-tcp.yml, and manifests/libvirt-tls.yml.
• Throttling capabilities. Slower disks become problematic for larger deployments. A simple file-locking mechanism throttles VM creation to not swamp slower hosts. See manifests/throttle-file-lock.yml. This capability is expandable if other needs arise.
• Ability to experiment with various stemcells. There are 3 stemcells which may work with Libvirt. See Stemcell experiments for details.

Libvirt Versions

Stemcell Versions

Status

The CPI is capable of running and managing BOSH deployments, generally without issue. Longer term management (snapshotting vms, for instance) either are untested or unimplemented at this point.

Feel free to try it out! Feedback and PR's are welcome.

Notes:

As of v3, disk management is configurable with throttling. This was done to help with deployments like Cloud Foundry that create 15 (or more!) VM's at once. With slower disks (non-NVMe) the machine may not be able to keep up. This keeps it manageable at the cost of slowing down VM creation.

As of v4, disk assignment allows for disk resizing. Disks as still assigned statically; current assignment scheme:

• /dev/vda: boot disk
• /dev/vdb: ephemeral disk
• /dev/vdc: config disk
• /dev/vdd or /dev/vde: persistent disk (optional); two expected when resizing disks

Tickets being watched

• Libvirt command switcheroo (work-around in place): ticket

Closed

• Blocking IaaS provided disk resize: ticket
• Libvirt enhancement request to allow network ingress to be specified via API: ticket
• Issues with TLS connections after a certain commit: ticket

Documentation

• Libvirt Configuration
• Setup the BOSH Director
• Useful Utilities
• Deploying Software to BOSH
• Development notes
• Stemcell experiments
• TODO list