search

shawnanastasio / libkvmchan

An implementation of the Xen vchan API on KVM
Other
10 stars 4 forks source link

Startup problems #17

Open pwmarcz opened 4 years ago

pwmarcz commented 4 years ago

Hi, @marmarek's friend here, I'm trying to get the code to run and understand it so that we can bring the Qubes on KVM effort forward.

I am stuck trying to get anything to run. Could you give me some pointers? Unfortunately I'm not very familiar with qemu and libvirt.

(I'll be happy to fix these issues once I understand what's happening!)

$ ./kvmchand 
[INFO] daemon/libvirt.c:530: Domain fedora(6, UUID: 63b6b932-f83a-48e7-9c0c-6eb91c908768) changed state (event: 4, detail: 0)!
[WARN] daemon/libvirt.c:604: Unknown lifecycle event 10! Ignoring...
[INFO] daemon/libvirt.c:530: Domain fedora(6, UUID: 63b6b932-f83a-48e7-9c0c-6eb91c908768) changed state (event: 2, detail: 0)!
[INFO] daemon/libvirt.c:262: About to attach ivshmem device at /tmp/kvmchand/ivshmem_socket, index 0
[INFO] daemon/ivshmem.c:534: Got connection from PID: 108108
[ERROR] daemon/libvirt.c:344: QEMU rejected ivshmem device: {"id":"libvirt-372","error":{"class":"GenericError","desc":"Bus 'pcie.0' does not support hotplugging"}}
[WARN] daemon/libvirt.c:552: Failed to attach ivshmem device! Ignoring...

I'm not sure how I should attach the device to work. I manually added "bus": "pci.1" to the QEMU request, and it seems to have done something, but kvmchand -g still fails with Unable to find ivshmem device.

[ERROR] daemon/libvirt.c:331: QEMU rejected chardev: {"id":"libvirt-396","error":{"class":"GenericError","desc":"attempt to add duplicate property 'charshmem0' to object (type 'container')"}}

It seems there is no graceful shutdown that would remove the devices from a VM? Or perhaps we should detect a situation where the devices have already been added.

nrgaway commented 4 years ago

I have it mostly working. I will update my repos within the next few hours. I will post the comments in #12.

shawnanastasio commented 4 years ago

To get up and running on x86_64 currently, the easiest way is to use an i440fx guest. Q35 guests will require some slight changes in both the guests's libvirt xml definition and libkvmchan's hotplug code. @nrgaway has been experimenting with this recently, and soon we should have Q35 guests working and documented.

In addition, you'll need to make sure the guest's kernel has CONFIG_VFIO_NOIOMMU set in order to use the guest-side daemon.

nrgaway commented 4 years ago

I pushed my changes to my repos. You can do a complete build for host and guest templates or just build
core-libvirt (kvm} core-vchan-libkvmchan (master) core-vchan-kvm (master) core-qubesdb (kvm} core-qrexec (kvm}

The following packages are required if you want to build all the Qubes packages for host and guest VM (template) kvm (master) linux-utils (kvm} core-admin (kvm} core-agent-linux (kvm} linux-kernel (vfio_noiommu)

And a builder.conf file may be helpful.

All the packages build and template will start using qvm-start but there is still a long way to go before everything is fully functional. Some manual configuration of the host has to be preformed.

I also include a libvirt XML configuration that works with libkvmchan and Qubes, taking note of the machine type and PCI controller configuration.

<domain type='kvm'>
  <name>fedora-32</name>
  <uuid>12345678-1234-1234-1234-123456789012</uuid>
  <memory unit='KiB'>4096000</memory>
  <currentMemory unit='KiB'>4096000</currentMemory>
  <vcpu placement='static'>2</vcpu>

  <-- MACHINE, UEFI BIOS -->
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/fedora-32_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='variable' adjustment='0' basis='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <-- DISK IMAGES -->
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-templates/fedora-32/root-dirty.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-templates/fedora-32/private-dirty.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-templates/fedora-32/volatile-dirty.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-kernels/5.6.16-1/modules.img'/>
      <target dev='vdd' bus='virtio'/>
      <readonly/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </disk>

    <-- PCI CONTROLLERS -->
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='pci' index='1' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>

    <-- CONTROLLER USED FOR LIBVCHAN (with patches alreaded pre-appliedin core-vchan-libkvmchan] -->
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>

    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>

    <-- HOST SHARES - REMOVE IF UNWANTED - ENABLED BY DEFAULT FOR DEVELOPMENT -->
    <filesystem type='mount' accessmode='squash'>
      <source dir='/home/kvm/share'/>
      <target dir='/share'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </filesystem>
    <filesystem type='mount' accessmode='squash'>
      <source dir='/home/kvm/qubes/qubes-packages-mirror-repo/vm-fc32/rpm'/>
      <target dir='/rpm'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </filesystem>

    <interface type='network'>
      <mac address='12:34:56:78:90:12'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

   <-- CONSOLE ACCESS, no gui --->
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>

    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </rng>
  </devices>
</domain>

Some manual VM configs may need to be performed depending on which Qubes packages you installed:

If you are brave enough to install the host components, only build for Fedora 32 Dom0 and VM and note I have only tested with BTRFS on the host since it's easier to debug and replace images when testing. Here is a list of manual configuration that is currently required.:

Hopefully some of this helps.

nrgaway commented 4 years ago

I also pushed post-installation steps after building Qubes-KVM which I'll keep updating as issues arise or resolved. I'm hoping to eliminate most, if not all, the VM post configuration tasks within the next week or so.

pwmarcz commented 4 years ago

Thanks! I was able to get the example for libkvmchan running with the help of your XML.

For now I'll be focusing on reading the base library, so I'm not installing Qubes yet, but the instructions will definitely come in handy later.