Convenience of containers, security of virtual machines

With firebuild, you can build and deploy secure VMs directly from Dockerfiles and Docker images in just few minutes.

The concept of firebuild is to leverage as much of the existing Docker world as possible. There are thousands of Docker images out there. Docker images are awesome because they encapsulate the software we want to run in our workloads, they also encapsulate dependencies. Dockerfiles are what Docker images are built from. Dockeriles are the blueprints of the modern infrastructure. There are thousands of them for almost anything one can imagine and new ones are very easy to write.

With firebuild it is possible to:

• build root file systems directly from Dockerfiles
• tag and version root file systems
• run and manage microvms on a single host
• define run profiles

High level example

Build and start HashiCorp Consul 1.9.4 on Firecracker with three simple steps:

• build a base operating system image
• build Consul image
• start the application

sudo $GOPATH/bin/firebuild baseos \
    --profile=standard \
    --dockerfile $(pwd)/baseos/_/alpine/3.12/Dockerfile

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile \
    --cni-network-name=machine-builds \
    --ssh-user=alpine \
    --vmlinux-id=vmlinux-v5.8 \
    --tag=combust-labs/consul:1.9.4

sudo $GOPATH/bin/firebuild run \
    --profile=standard \
    --name=consul1 \
    --from=combust-labs/consul:1.9.4 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8

Find the IP of the consul1 VM and query Consul:

VMIP=$(sudo $GOPATH/bin/firebuild inspect \
    --profile=standard \
    --vmm-id=consul1 | jq '.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IP' -r)

$ curl http://${VMIP}:8500/v1/status/leader
"127.0.0.1:8300"

But how?

clone and build from sources

mkdir -p $GOPATH/src/github.com/combust-labs/firebuild
cd $GOPATH/src/github.com/combust-labs/firebuild
go install

The binary will be placed in $GOPATH/bin/firebuild.

create a profile

# create required directories, these need to exist before the profile can be created:
sudo mkdir -p /firecracker/rootfs
sudo mkdir -p /firecracker/vmlinux
sudo mkdir -p /srv/jailer
sudo mkdir -p /var/lib/firebuild
# create a profile:
sudo $GOPATH/bin/firebuild profile-create \
    --profile=standard \
    --binary-firecracker=$(readlink /usr/bin/firecracker) \
    --binary-jailer=$(readlink /usr/bin/jailer) \
    --chroot-base=/srv/jailer \
    --run-cache=/var/lib/firebuild \
    --storage-provider=directory \
    --storage-provider-property-string="rootfs-storage-root=/firecracker/rootfs" \
    --storage-provider-property-string="kernel-storage-root=/firecracker/vmlinux" \
    --tracing-enable

Kernel images will be stored in /firecracker/vmlinux, root file systems will be stored in /firecracker/rootfs.

build the kernel

The examples use the 5.8 Linux kernel image which is built using the configuration from the baseos/kernel/5.8.config file in this repository. To build the kernel:

export KERNEL_VERSION=v5.8
mkdir -p /tmp/linux && cd /tmp/linux
git clone https://github.com/torvalds/linux.git .
git checkout ${KERNEL_VERSION}
wget -O .config https://raw.githubusercontent.com/combust-labs/firebuild/master/baseos/kernel/5.8.config
make vmlinux -j32 # adapt to the number of cores you have

Once built, copy the kernel to the storage:

mv /tmp/linux/vmlinux /firecracker/vmlinux/vmlinux-${KERNEL_VERSION}

setup CNI

firebuild assumes CNI availability. Installing the plugins is very straightforward. Create /opt/cni/bin/ directory and download the plugins:

mkdir -p /opt/cni/bin
curl -O -L https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz
tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v0.9.1.tgz

Firecracker also requires the tc-redirect-tap plugin. Unfortunately, this one does not offer downloadable binaries and has to be built from sources.

mkdir -p $GOPATH/src/github.com/awslabs/tc-redirect-tap
cd $GOPATH/src/github.com/awslabs/tc-redirect-tap
git clone https://github.com/awslabs/tc-redirect-tap.git .
make install

create a dedicated CNI network for the builds

Feel free to change the ipam.subnet or set multiple ones. host-local IPAM CNI plugin documentation.

cat <<EOF > /etc/cni/conf.d/machine-builds.conflist
{
    "name": "machine-builds",
    "cniVersion": "0.4.0",
    "plugins": [
        {
            "type": "bridge",
            "name": "builds-bridge",
            "bridge": "builds0",
            "isDefaultGateway": true,
            "ipMasq": true,
            "hairpinMode": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.128.0/24",
                "resolvConf": "/etc/resolv.conf"
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "tc-redirect-tap"
        }
    ]
}
EOF

caution

The maximum socket path in the Linux Kernel is 107 characters + \0:

struct sockaddr_un {
    __kernel_sa_family_t sun_family; /* AF_UNIX */
    char sun_path[UNIX_PATH_MAX];   /* pathname */
};

The --chroot-base value must have a maximum length of 31 characters. The constant jailer path suffix used by firebuild is 76 characters:

• constant /firecracker-v0.22.4-x86_64/ (automatically generated by the jailer)
• VM ID is always 20 characters long
• constant /root/run/firecracker.socket assumed by the jailer

Example: /firecracker-v0.22.4-x86_64/sifuqm4rq2runxparjcx/root/run/firecracker.socket.

Using more than 31 characters for the --chroot-base value, regardless if in the profile setting or using the command --chroot-base flag, will lead to a very obscure error. Firecracker will report an error similar to:

INFO[0006] Called startVMM(), setting up a VMM on /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket
WARN[0009] Failed handler "fcinit.StartVMM": Firecracker did not create API socket /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket: context deadline exceeded
{"@level":"error","@message":"Firecracker VMM did not start, build failed","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856228Z","reason":"Failed to start machine: Firecracker did not create API socket /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket: context deadline exceeded","veth-name":"vethHvfZiskhLkQ","vmm-id":"6b41ecc3783c4f38a743c9c8af4bbe0f"}
{"@level":"info","@message":"cleaning up jail directory","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856407Z","veth-name":"vethHvfZiskhLkQ","vmm-id":"6b41ecc3783c4f38a743c9c8af4bbe0f"}
{"@level":"info","@message":"cleaning up temp build directory","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856458Z"}
WARN[0010] firecracker exited: signal: killed

In the above example, the path is 114 characters long. Changing the chroot to /mnt/sdd1/fc/jail would solve the problem.

build the base operating system root file system

firebuild uses the Docker metaphor. An image of an application is built FROM a base. An application image can be built FROM alpine:3.13, for example. Or FROM debian:buster-slim, or FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3 and dozens others.

In order to fulfill those semantics, a base operating system image must be built before the application root file system can be created.

Build a base Debian Buster slim:

sudo $GOPATH/bin/firebuild baseos \
    --profile=standard \
    --dockerfile $(pwd)/baseos/_/debian/buster-slim/Dockerfile

Because the baseos root file system is built completely with Docker, there is no need to configure the kernel storage.

This does not belong here, structure better: It's possible to tag the baseos output using the --tag= argument, for example:

sudo $GOPATH/bin/firebuild baseos \
    --profile=standard \
    --dockerfile $(pwd)/baseos/_/debian/buster-slim/Dockerfile \
    --tag=custom/os:latest

create a Postgres 13 VM rootfs directly from the upstream Dockerfile

The upstream Dockerfile is built FROM debian:buster-slim, that's the baseos built in the previous step:

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --dockerfile=git+https://github.com/docker-library/postgres.git:/13/Dockerfile \
    --cni-network-name=machine-builds \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --tag=combust-labs/postgres:13

create a separate CNI network for running VMs

For example:

cat <<EOF > /etc/cni/conf.d/machines.conflist
{
    "name": "machines",
    "cniVersion": "0.4.0",
    "plugins": [
        {
            "type": "bridge",
            "name": "machines-bridge",
            "bridge": "machines0",
            "isDefaultGateway": true,
            "ipMasq": true,
            "hairpinMode": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.127.0/24",
                "resolvConf": "/etc/resolv.conf"
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "tc-redirect-tap"
        }
    ]
}
EOF

run the VM from the resulting tag

Once the root file system is built, start the VM:

sudo $GOPATH/bin/firebuild run \
    --profile=standard \
    --name=postgres1 \
    --from=combust-labs/postgres:13 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --env="POSTGRES_PASSWORD=some-password"

To avoid passing the password on the command line, you can use --env-file flag instead. The database is running, to verify:

Fine the IP address of the Postgres VM:

VMIP=$(sudo $GOPATH/bin/firebuild inspect \
    --profile=standard \
    --vmm-id=postgres1 | jq '.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IP' -r)

$ nc -zv ${VMIP} 5432
Connection to 192.168.127.94 5432 port [tcp/postgresql] succeeded!

If SSH access to the VM is required, this command can be used instead:

sudo $GOPATH/bin/firebuild run \
    --profile=standard \
    --name=postgres2 \
    --from=combust-labs/postgres:13 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --env="POSTGRES_PASSWORD=some-password" \
    --ssh-user=debian \
    --identity-file=path/to/the/identity.pub

additional run flags

• --daemonize: when specified, runs the VM in a daemonized mode
• --env-file: full path to the environment file, multiple OK
• --env: environment variable to deploy to configure the VM with, multiple OK, format --env=VAR_NAME=value
• --hostname: hostname to apply to the VM which the VM uses to resolve itself
• --name: name of the virtual machine, if empty, random string will be used, maxmimum 20 characters, only a-zA-Z0-9 ranges are allowed
• --ssh-user: username to get access to the VM via SSH with, these are defined in the baseos Dockerfiles and follow the EC2 pattern: alpine for Alpine images and debian for Debian image; together with --identity-file allows access to the running VM via SSH
• --identity-file: full path to the publish SSH key to deploy to the running VM

environment merging

The final environment variables are written to /etc/profile.d/run-env.sh file. All files specified with --env-file are merged first in the order of occurrcence, variables specified with --env are merged last.

build directly from a Docker image

Sometimes having just the Dockerfile is not sufficient to execute a rootfs build. A good example is this Jaeger all-in-one Dockerfile. The Dockerfile depends on the binary artifact built via Makefile prior to Docker build. In this case, it's possible to build the VM rootfs directly from the Docker image:

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --docker-image=jaegertracing/all-in-one:1.22 \
    --docker-image-base=alpine:3.13 \
    --cni-network-name=machine-builds \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --tag=combust-labs/jaeger-all-in-one:1.22

The --docker-image-base is required because the underlying operating system the image was built from cannot be established from the Docker manifest.

To access the Jaeger Query UI via the host:

sudo iptables -t filter -A FORWARD \
    -m comment --comment "jaeger:1.22" \
    -p tcp -d 192.168.127.100 --dport 16686 \
    -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -t nat -A PREROUTING \
    -m comment --comment "jaeger:1.22" \
    -p tcp -i eno1 --dport 16686 \
    -j DNAT \
    --to-destination 192.168.127.100:16686

Where the exact IP address can be obtained using the firebuild inspect --profile=... --vmm-id=... command and the destination IP and interface depend on your configuration, you can use ip link to find the up broadcast interfaces and relevant IP address. Tool intergration will be added at a later stage.

how does it work

The builder pulls the requested Docker image with Docker. It then open the Docker image via the Docker save command and looks up the manifest.json and the Docker image config json explicitly stated in the manifest. When config is fetched, a temporary Dockerfile is built from the Docker config history. Any ADD and COPY commands for resources other than first / are used to extract files from the saved source image. When resources are exported, the build further continues exactly the same way as in case of the Dockerfile build.

terminating a daemonized VM

A VM started with the --daemonize flag can be stopped in three ways:

• by executing the kill tool command, this is a clean stop which will take care of all the necessry clean up
• by executing reboot from inside of the VM SSH connection; unclean stop, manual purge of the CNI cache, jailer directory, run cache and the veth link is needed
• by executing the cURL HTTP against the VM socket file; unclean stop, manual purge of the CNI cache, jailer directory, run cache and the veth link is needed

VM kill command

To get the VM ID, look closely at the output of the run ... --detached command:

{
    "@level":"info",
    "@message":"VMM running as a daemon",
    "@module":"run",
    "@timestamp":"2021-03-09T19:55:41.684488Z",
    "cache-dir":"/var/lib/firebuild/831b7068f7924584b384260e8d262834",
    "ip-address":"192.168.127.3",
    "ip-net":"192.168.127.3/24",
    "jailer-dir":"/srv/jailer/firecracker-v0.22.4-x86_64/831b7068f7924584b384260e8d262834",
    "pid":17904,
    "veth-name":"vethydMSApKfoDu",
    "vmm-id":"831b7068f7924584b384260e8d262834"
}

Copy the VM ID from the output and run:

sudo $GOPATH/bin/firebuild kill --profile=standard --vmm-id=${VMMID}

purging the remains of the VMs stopped without the kill command

If a VM exits in any other way than via kill command, following data continues residing on the host:

• jail directory with all contents
• run cache directory with all contents
• CNI interface with CNI cache directory

To remove this data, run the purge command.

sudo $GOPATH/bin/firebuild purge --profile=standard

list VMs

sudo $GOPATH/bin/firebuild ls --profile=standard

Example output:

2021-03-12T01:46:21.752Z [INFO]  ls: vmm: id=df45b6e14538456286e4a4bc1f9bf6e2 running=true pid=20658 image=tests/postgres:13 started="2021-03-12 01:46:11 +0000 UTC" ip-address=192.168.127.9

Dockerfile git+http(s):// URL

It's possible to reference a Dockerfile residing in the git repository available under a HTTP(s) URL. Here's an example:

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile#master \
    --cni-network-name=machine-builds \
    --vmlinux-id=vmlinux-v5.8 \
    --tag=combust-labs/consul:1.9.4

The URL format is:

git+http(s)://host:port/path/to/repo.git:/path/to/Dockerfile[#<commit-hash | branch-name | tag-name>]

And will be processed as:

• path /path/to/repo.git:/path/to/Dockerfile will be split by : and must contain both sides
  • /path/to/repo.git is the git repository path
  • /path/to/Dockerfile is the path to the Dockerfile in the repository, must point to a file after clone and checkout
• optional #fragment may be a comit hash, a branch name or a tag name
  • if no #fragment is given, the program will use the default cloned branch, check the remote to find out what is it
• the cloned repository will have a single remote and the first remote wil be used

supported Dockerfile URL formats

• http:// and https:// for direct paths to the Dockerfile, these can handle single file only and do not attempt loading any resources handled by ADD / COPY commands, the server must be capable of responding to HEAD and GET http requests, more details in Caveats when building from the URL further in this document
• special git+http:// and git+https://, documented above
• standard ssh://, git:// and git+ssh:// URL formats with the expectation that the path meets the criteria from the git+http(s):// URL section above

caveats when building from the URL

The build command will resolve the resources referenced in ADD and COPY commands even when loading the Dockerfile via the URL. The context root in this case will be established by removing the file name from the URL. An example:

• consider the URL https://raw.githubusercontent.com/hashicorp/docker-consul/master/0.X/Dockerfile
• the Dockerfile name will be removed from the URL and the context is https://raw.githubusercontent.com/hashicorp/docker-consul/master/0.X
• assuming that the Dockerfile contains ADD ./docker-entrypoint.sh ..., the resolver will try loading https://raw.githubusercontent.com/hashicorp/docker-consul/master/0.X/docker-entrypoint.sh

There are following limitations when loading the resources like that via URL:

• if the ADD or COPY points to a directory, the command will fail because there is no unified way of loading directories via HTTP, the resolver will not even attempt this, it will most likely fail on the HTTP GET request
• the file permissions will not be carried over because there is no method to infer file mode from a HTTP response

unsupported Dockerfile features

The build program does not support:

• ONBUILD commands
• HEALTHCHECK commands
• STOPSIGNAL commands

multi-stage Dockerfile builds

firebuild supports multi-stage Dockerfile builds. An example with grepplabs Kafka Proxy.

Build v0.2.8 using git repository link:

sudo $GOPATH/bin/firebuild rootfs \
    --profile=standard \
    --dockerfile=git+https://github.com/grepplabs/kafka-proxy.git:/Dockerfile#v0.2.8 \
    --cni-network-name=machine-builds \
    --vmlinux-id=vmlinux-v5.8 \
    --tag=combust-labs/kafka-proxy:0.2.8

tracing

TODO: eat your own dog food, start with firebuild.

Start Jaeger, for example:

docker run --rm -ti \
    -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one:1.22

And configure respective commands with:

... --tracing-enable \
--tracing-collector-host-port=... \

The default value of the --tracing-collector-host-port is 127.0.0.1:6831. To enable tracer log output, set --tracing-log-enable flag.

license

Unless explcitly stated: AGPL-3.0 License.