Use case documentation needed: provisioning VM using libvirt

[Vanuan]

What do I need first? Do I need to write a playbook?

I'm looking to setup something like this:

It looks like currently infrakit supports CLI-based deployment using playbooks. Do I get it right?

So if I somehow create a playbook, I would be able to provision a libvirt-based instance?

[Vanuan]

First of all, I think I need to start libvirt plugin, right? But it looks like it isn't available for some reason:

infrakit plugin start
INFO[10-11|04:02:06] config                                   module=cli/plugin url= fn=github.com/docker/infrakit/cmd/infrakit/plugin.Command.func2
Plugins available:
KIND                    EXEC
aws                     inproc
combo                   inproc
digitalocean            inproc
docker                  inproc
enrollment              inproc
file                    inproc
gc                      inproc
google                  inproc
group                   inproc
hyperkit                inproc
ibmcloud                inproc
image                   inproc
ingress                 inproc
inventory               inproc
kubernetes              inproc
maas                    inproc
manager                 inproc
oneview                 inproc
oracle                  inproc
packet                  inproc
pool                    inproc
rackhd                  inproc
resource                inproc
selector/spread         inproc
selector/tiered         inproc
selector/weighted       inproc
simulator               inproc
swarm                   inproc
tailer                  inproc
terraform               inproc
time                    inproc
vagrant                 inproc
vanilla                 inproc
vars                    inproc
vsphere                 inproc

This led me to assuming that this function didn't run for some reason: https://github.com/docker/infrakit/blob/a10d75ee2d0a8992edf415630a47b0633b04b0b0/pkg/run/v0/libvirt/libvirt.go#L33-L35

[Vanuan]

After trying to set a breakpoint it says No source file named pkg/run/v0/libvirt/libvirt.go.

So it means it isn't imported anywhere, right?

So I went to see where all providers are imported and look what I found:

https://github.com/docker/infrakit/blob/a10d75ee2d0a8992edf415630a47b0633b04b0b0/cmd/infrakit/providers.go#L5-L22

Git blame led to this: https://github.com/docker/infrakit/pull/872 So libvirt isn't built by default?

Looking back found this: https://github.com/docker/infrakit/pull/718 Along with instructions on how to build it with libvirt.

Apparently there are some issues with go buildings for libvirt. README of this package says it requires libvirt-dev package installed. And in runtime it requires libvirt.so even if you don't use it. Authors suggest to provide libvirt as a separate binary: https://github.com/libvirt/libvirt-go/issues/8

It probably makes sense to make plugins/providers actually installable as separate binaries. Not sure whether it is designed that way.

[Vanuan]

So, this is how I build infrakit with libvirt provider:

sudo apt-get install libvirt-dev
GO_BUILD_TAGS="libvirt builtin providers" make build/infrakit

And it works:

$ infrakit plugin start libvirt
...
$ infrakit plugin ls
INTERFACE           NAME                          LISTEN
Instance/0.6.1      libvirt/default               /home/john/.infrakit/plugins/libvirt

Now I need to figure out how to provide libvirt URI and then - how to start some instances, and then - how to configure those instances.

[Vanuan]

It was quite hard to find. To provide libvirt URIs we use this:

https://github.com/docker/infrakit/blob/a10d75ee2d0a8992edf415630a47b0633b04b0b0/pkg/run/v0/libvirt/libvirt.go#L26

For example:

$ export INFRAKIT_LIBVIRT_URIS=default=qemu+ssh://server/system
$ infrakit plugin start libvirt

To start instances the command line is this:

$ infrakit local libvirt/default provision -y libvirt.yml

Where libvirt.yml looks like this:

Properties:
  MetadataStoragePool: default
  Domain:
  Domain:
    Type: kvm
    Memory:
      Unit: GiB
      Value: 1
    VCPU:
      Value: 1
    OS:
      Type:
        Arch: x86_64
        Machine: pc-i440fx-2.8
        Type: hvm
      Initrd: ./initrd.img
      Kernel: ./bzImage
      KernelArgs: console=ttyS0
      BIOS:
        UseSerial: 'yes'
        RebootTimeout: '1000'
      Boot:
        Dev: hd
    Devices:
      Interfaces:
      - Type: bridge
        Source:
          Bridge: virbr0
        Model:
          Type: virtio
      Serials:
      - Type: pty
      Consoles:
      - Type: pty
        Target:
          Type: serial
          Name: "0"
      Channels:
      - Type: unix
        Target:
          Type: virtio
          Name: org.qemu.guest_agent.0
    OnCrash: destroy
    OnPoweroff: destroy
    OnReboot: restart

Now I need to figure out how to generate those kernel and initrd files.

[Vanuan]

It looks like linuxkit can generate those files. Installed linuxkit like this:

wget https://github.com/linuxkit/linuxkit/releases/download/v0.6/linuxkit-linux-amd64 -O ~/bin/linuxkit
chmod +x ~/bin/linuxkit

Running it like this

linuxkit build --format kernel+initrd linuxkit.yml

where linuxkit is here:

https://github.com/linuxkit/linuxkit/blob/master/linuxkit.yml

Unfortunately, kernel 4.14.74 causes kernel panic when running in qemu. So I switched to 4.4.159

Now I need to solve the next problem: how to build those images or copy them to the remote libvirtd host (when using INFRAKIT_LIBVIRT_URIS=default=qemu+ssh://server/system). Maybe a simple NFS share for now?

[thebsdbox]

We could perhaps extend the functionality from here https://github.com/docker/infrakit/tree/master/pkg/x/remoteboot ?

[Vanuan]

Not sure I understand. remoteboot looks like a PXE server. How does it help?

Currently I'm struggling to find steps needed to provision kvm instances according to some yaml file. And configure those instances with docker swarm/kubernetes.

Here's a list of problems I found along the way (some of them solved):

1. libvirt is not builtin to infrakit
2. providing libvirt remote URI is not documented (INFRAKIT_LIBVIRT_URIS)
3. example libvirt.yaml doesn't work
4. you have to build kernel and initrd using thirdparty tool (at least linuxkit integration is not documented)
5. you have to copy kernel and initrd on remote host is using remote libvirt URI (at least didn't find a way to copy automatically)
6. video device is not working for some reason
7. can't figure out how to specify random device (https://github.com/libvirt/libvirt-go-xml/issues/25)
8. init section doesn't work (something with permissions for "metadata volume")
9. figuring out networking (should I use portforwarding, ip routing or something else to reach VMs)
10. what's linuxkit configuration to build images with docker installed
11. how to setup container orchestration (say, I want 3 manager and 7 worker VMs), what flavors/groups should I use
12. should I reach container orchestrator directly or through infrakit
13. should I install loadbalancer on metal, VM or container

I'm currently on step 7. It looks like something to do with libvirtxml (https://github.com/libvirt/libvirt-go-xml/issues/25)

[Vanuan]

BTW VNC instead of video device working fine (problem 6):

      Graphics:
      - Type: "vnc"
        AutoPort: "yes"
        Port: -1
        Listen: "127.0.0.1"
        Listeners:
          - Type: "address"
            Address: "127.0.0.1"

[Vanuan]

Ok, the problem with RNG is that vendored copy isn't updated for more than a year: https://github.com/libvirt/libvirt-go-xml/commit/7ea44658ebbc28b44d53372502137ecd60caa975