Libvirt Resources Created by ViNO Lose Persistence on K8s Restart

lb4368 commented 3 years ago

Describe the bug After applying a ViNO CR that successfully creates VMs via vino-builder and then restarting k8s, the vino-builder container fails restart and the libvirt resources created previously are no longer persistent libvirt resources.

Steps To Reproduce

In a k8s cluster running ViNO operator, apply a ViNO CR which will build one or more VMs

Verify successful creation of libvirt resources:


$ k exec -i -t ds/default-vino-test-cr -c vino-builder -n vino-system -- /bin/bash
root@mocdtl03lb4368:/# virsh
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands 'quit' to quit

virsh # pool-info vino-default Name: vino-default UUID: b0290365-80dd-492e-80bf-d7f3d6e77a72 State: running Persistent: yes Autostart: yes Capacity: 28.00 GiB Allocation: 22.86 GiB Available: 5.14 GiB

virsh # net-info management Name: management UUID: f92188ca-9fc1-4873-89bf-51c9fed3b21d Active: yes Persistent: yes Autostart: yes Bridge: management

virsh # dominfo master-0 Id: 1 Name: master-0 UUID: 9d9bf759-a5b9-7729-ffa7-d277425bba0a OS Type: hvm State: running CPU(s): 1 CPU time: 2.1s Max memory: 4194304 KiB Used memory: 4194304 KiB Persistent: yes Autostart: enable Managed save: no Security model: none Security DOI: 0

virsh # dominfo worker-0 Id: 2 Name: worker-0 UUID: 2f5d8d23-6660-d059-1e48-7847bd050202 OS Type: hvm State: running CPU(s): 1 CPU time: 2.0s Max memory: 2097152 KiB Used memory: 2097152 KiB Persistent: yes Autostart: enable Managed save: no Security model: none

- Restart k8s
- Note error in vino-builder container log:

TASK [libvirt : define the storage pool] *** fatal: [localhost]: FAILED! => {"changed": true, "cmd": "virsh pool-define /tmp/storage-vino-default.xml", "delta": "0:00:00.028030", "end": "2021-04-09 16:51:49.029641", "msg": "non-zero return code", "rc": 1, "start": "2021-04-09 16:51:49.001611", "stderr": "error: Failed to define pool from /tmp/storage-vino-default.xml\nerror: operation failed: pool 'vino-default' already exists with uuid b0290365-80dd-492e-80bf-d7f3d6e77a72", "stderr_lines": ["error: Failed to define pool from /tmp/storage-vino-default.xml", "error: operation failed: pool 'vino-default' already exists with uuid b0290365-80dd-492e-80bf-d7f3d6e77a72"], "stdout": "", "stdout_lines": []}

- Note that libvirt resources are no longer marked as persistent and no longer have autostart enabled:

virsh # pool-info vino-default Name: vino-default UUID: b0290365-80dd-492e-80bf-d7f3d6e77a72 State: running Persistent: no Autostart: no Capacity: 28.00 GiB Allocation: 22.86 GiB Available: 5.13 GiB

virsh # net-info management Name: management UUID: f92188ca-9fc1-4873-89bf-51c9fed3b21d Active: yes Persistent: no Autostart: no Bridge: management

virsh # dominfo master-0 Id: 1 Name: master-0 UUID: 9d9bf759-a5b9-7729-ffa7-d277425bba0a OS Type: hvm State: running CPU(s): 1 CPU time: 6.5s Max memory: 4194304 KiB Used memory: 4194304 KiB Persistent: no Autostart: disable Managed save: no Security model: none Security DOI: 0

virsh # dominfo worker-0 Id: 2 Name: worker-0 UUID: 2f5d8d23-6660-d059-1e48-7847bd050202 OS Type: hvm State: running CPU(s): 1 CPU time: 6.3s Max memory: 2097152 KiB Used memory: 2097152 KiB Persistent: no Autostart: disable Managed save: no Security model: none Security DOI: 0



**Expected behavior**
The vino-builder container should restart without error and libvirt resources should retain their persistence/autostart.

**Environment**
- vino controller Version:  https://github.com/airshipit/vino/commit/c6c43fa1611ced878f7d6ddadbc728ccd4300551
- Operating System:  Ubuntu 20.04
- Kernel version:
- Kubernetes Version:  1.19.6
- Go version:
- Hardware specs (e.g. 4 vCPUs, 16GB RAM, bare metal vs VM):  VirtualBox

lb4368 commented 3 years ago

There will also be a future requirement that these libvirt resources survive Day 2 operations such as a re-image of the host kubernetes node. Would a fix for this bug be related?

teoyaomiqui commented 3 years ago

related patchset https://review.opendev.org/c/airship/images/+/785688 was merged. @lb4368 please verify that issue can be closed

teoyaomiqui commented 3 years ago

@lb4368 can u paste dump xml for these objects and paste them here as well?

lb4368 commented 3 years ago

@teoyaomiqui Here are the xml dumps for resources that were created with the sample ViNO CR after your patchset was merged. Resources are still initially being created as persistent, but after k8s restart the resources exist but are no longer marked as persistent.

virsh # pool-dumpxml vino-default
<pool type='dir'>
  <name>vino-default</name>
  <uuid>f230afd5-cbb9-4453-94b1-64e6518255a5</uuid>
  <capacity unit='bytes'>30059413504</capacity>
  <allocation unit='bytes'>25411600384</allocation>
  <available unit='bytes'>4647813120</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/vino</path>
    <permissions>
      <mode>0711</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

virsh # vol-dumpxml master-0 --pool vino-default
<volume type='file'>
  <name>master-0</name>
  <key>/var/lib/libvirt/vino/master-0</key>
  <source>
  </source>
  <capacity unit='bytes'>32212254720</capacity>
  <allocation unit='bytes'>200704</allocation>
  <target>
    <path>/var/lib/libvirt/vino/master-0</path>
    <format type='qcow2'/>
    <permissions>
      <mode>0600</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
    <timestamps>
      <atime>1618933488.004627894</atime>
      <mtime>1618932796.693321839</mtime>
      <ctime>1618932796.701321878</ctime>
    </timestamps>
  </target>
</volume>

virsh # vol-dumpxml worker-0 --pool vino-default
<volume type='file'>
  <name>worker-0</name>
  <key>/var/lib/libvirt/vino/worker-0</key>
  <source>
  </source>
  <capacity unit='bytes'>10737418240</capacity>
  <allocation unit='bytes'>200704</allocation>
  <target>
    <path>/var/lib/libvirt/vino/worker-0</path>
    <format type='qcow2'/>
    <permissions>
      <mode>0600</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
    <timestamps>
      <atime>1618933488.004627894</atime>
      <mtime>1618932801.497345170</mtime>
      <ctime>1618932801.509345229</ctime>
    </timestamps>
  </target>
</volume>

virsh # dumpxml master-0
<domain type='kvm'>
  <name>master-0</name>
  <uuid>9d9bf759-a5b9-7729-ffa7-d277425bba0a</uuid>
  <metadata/>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>8192</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <emulatorpin cpuset='2'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source pool='vino-default' volume='master-0'/>
      <target dev='vde' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='0a:00:00:00:00:00'/>
      <source bridge='external'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='02:00:00:00:00:00'/>
      <source bridge='management'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>

virsh # dumpxml worker-0
<domain type='kvm'>
  <name>worker-0</name>
  <uuid>2f5d8d23-6660-d059-1e48-7847bd050202</uuid>
  <metadata/>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>8192</shares>
    <vcpupin vcpu='0' cpuset='3'/>
    <emulatorpin cpuset='3'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source pool='vino-default' volume='worker-0'/>
      <target dev='vde' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='0a:00:00:00:00:01'/>
      <source bridge='external'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='02:00:00:00:00:01'/>
      <source bridge='management'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>

lb4368 commented 3 years ago

I tried another test where after applying the ViNO CR, I restarted the machine as opposed to just restarting K8s. In this case, the libvirt resources are all recreated with persistent=yes.

virsh # pool-info vino-default
Name:           vino-default
UUID:           3e3c003a-27f0-422c-9d21-192cedb32a4a
State:          running
Persistent:     yes
Autostart:      no
Capacity:       28.00 GiB
Allocation:     23.57 GiB
Available:      4.43 GiB

airshipbot commented 3 years ago

Related Change #787437

Subject: Make storage pool persistent Link: https://review.opendev.org/c/airship/vino/+/787437 Status: MERGED Owner: Kostyantyn Kalynovskyi (kkalynovskyi@mirantis.com)

This change will close this issue when merged.

Approvals

Code-Review
+1 Lawrence Borgmeyer
+2 Ian Howell
+2 Matt McEuen
Verified
+2 Zuul
Workflow
+1 Matt McEuen

Last Updated: 2021-04-22 16:24:33 CDT

lb4368 commented 3 years ago

@teoyaomiqui I validated patchset https://review.opendev.org/c/airship/vino/+/787437. This issue can be closed once that patchset is merged. Thanks!

airshipit / vino

Libvirt Resources Created by ViNO Lose Persistence on K8s Restart #4

Related Change #787437

Approvals