hyperhq / runv

Hypervisor-based Runtime for OCI
Apache License 2.0
828 stars 129 forks source link

qemu driver vsock support #411

Closed bergwolf closed 7 years ago

bergwolf commented 7 years ago

The PR rebases https://github.com/hyperhq/runv/pull/355 on top of latest master. It changes qemu and libvirt driver to support vsock device in guest vm:

  1. try to load vsock host kernel module at startup
  2. if vsock host modules are supported and vsock is enabled via config, add vsock device to guest vm and create control and tty channels via vsock
  3. persistent vsock cid and load it at startup
[hypervsock@~]$sudo hyperctl run -d bergwolf/ubuntu-fio
POD id is ubuntu-fio-3551741209
Time to run a POD is 4477 ms

[hypervsock@~]$pa|grep vsock
root     128768  3.3  6.1 1368832 109172 ?      Sl   16:37   0:07 /usr/bin/qemu-system-x86_64 -machine pc-i440fx-2.1,accel=kvm,usb=off -global kvm-pit.lost_tick_policy=discard -cpu host -kernel /var/lib/hyper/kernel -initrd /var/lib/hyper/hyper-initrd.img -append console=ttyS0 panic=1 no_timer_check -realtime mlock=off -no-user-config -nodefaults -no-hpet -rtc base=utc,driftfix=slew -no-reboot -display none -boot strict=on -m size=64,slots=1,maxmem=32768M -smp cpus=1,maxcpus=8 -numa node,nodeid=0,cpus=0-7,mem=64 -qmp unix:/var/run/hyper/vm-HAvUnepiTO/qmp.sock,server,nowait -serial unix:/var/run/hyper/vm-HAvUnepiTO/console.sock,server,nowait -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -chardev socket,id=charch0,path=/var/run/hyper/vm-HAvUnepiTO/hyper.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/hyper/vm-HAvUnepiTO/tty.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -fsdev local,id=virtio9p,path=/var/run/hyper/vm-HAvUnepiTO/share_dir,security_model=none -device virtio-9p-pci,fsdev=virtio9p,mount_tag=share_dir -daemonize -pidfile /var/run/hyper/vm-HAvUnepiTO/pidfile -D /var/log/hyper/qemu/vm-HAvUnepiTO.log -device vhost-vsock-pci,id=vsock0,bus=pci.0,addr=5,guest-cid=1024

root@ubuntu-fio-3551741209:/# ./nc-vsock -l 1234
Connection from cid 2 port 1158...

Requires: https://github.com/hyperhq/hyperstart/pull/249

TODOs:

  1. need to handle vsock cid increment for runv cmd
  2. need to figure out how to make it work with template
bergwolf commented 7 years ago

@laijs updated with following requested changes:

  1. use vsock:// and unix:// to denote vsock and unix socket addresses, and convert utils/libhyperstart to handle both
  2. move vsock guest cid to vmcontext and enable vsock support for libvirt
  3. rename VsockCid to VsockCidAllocator
  4. move vsock dialer to vsock_conn.go

One thing to note:

    libvirt by default uses cgroup controller to limit and grant guest
    permissions. However, when adding device via qemu monitor directly,
    libvirt has no knowledge and thus does not grant proper permissions to
    qemu process. As a result, qemu will get EPERM opening /dev/vhost-vsock
    device.

    To work around it, remove devices from cgroup_controllers in
    /etc/libvirt/qemu.conf.