wking
commented
6 years ago I hit this too, and am trying to track down where the /var/lib/libvirt/images is coming from. On my host:
$ sudo ls /var/lib/libvirt/images/
$ virsh -c qemu+tcp://192.168.122.1/system pool-list
Name State Autostart
-------------------------------------------
default active yes
$ virsh -c qemu+tcp://192.168.122.1/system pool-dumpxml default
<pool type='dir'>
<name>default</name>
<uuid>c20a2154-aa60-44cf-bf37-cd8b7818a4e4</uuid>
<capacity unit='bytes'>105554829312</capacity>
<allocation unit='bytes'>44038131712</allocation>
<available unit='bytes'>61516697600</available>
<source>
</source>
<target>
<path>/home/trking/VirtualMachines</path>
<permissions>
<mode>0777</mode>
<owner>114032</owner>
<group>114032</group>
<label>system_u:object_r:virt_image_t:s0</label>
</permissions>
</target>
</pool>
$ ls /home/trking/VirtualMachines/
bootstrap bootstrap.ign coreos_base master0 master-0.ign worker.ignIn the installer, we have settings for the pool and volume, which we currently hard-code to default and coreos_base. We set the volume, so hard-coding that shouldn't be a problem. We don't set pool when we create the volume, so we get the Terraform provider's default default. So far, so good. We push those values into the cluster since #205 for the machine-config-operator to pick up (openshift/machine-config-operator#47). Our ImagePool and ImageVolume settings are recent (#271), but the MCO doesn't seem to be looking at either the old QCOWImagePath or the new Image* properties (at least as of openshift/machine-config-operator@d948fb8baa63a).
Then the chain of custody gets fuzzy for me.
On the other end, the /var/lib/... path is the actuator default for baseVolumePath, but I'm not clear on whether baseVolumePath plays into our chain.
From your logged:
I0921 18:24:47.612648 1 logs.go:41] [DEBUG] Create a libvirt volume with name worker-b9f4v for pool default from the base volume /var/lib/libvirt/images/coreos_basewe see that by the time we got here, we had default as the poolName (correct) and /var/lib/libvirt/images/coreos_base as the baseVolumeID (questionable). Then we lookup the pool. And then we die trying to find the volume in that pool. We should be looking up the volume by coreos_base; e.g. with virsh:
$ virsh -c qemu+tcp://192.168.122.1/system vol-info --vol coreos_base --pool default
Name: coreos_base
Type: file
Capacity: 16.00 GiB
Allocation: 1.55 GiB
The issue is probably the fact that that lookup also works when you happen to use the correct full path:
$ virsh -c qemu+tcp://192.168.122.1/system vol-info --vol /home/trking/VirtualMachines/coreos_base --pool default
Name: coreos_base
Type: file
Capacity: 16.00 GiB
Allocation: 1.55 GiB
So can we connect the dots between our config and the busted baseVolumeID? The actuator is getting the value from machine-provider config. Who writes that config? Maybe the machine-API operator using this template? A short-term patch is probably updating that template to use just coreos_base. A long-term fix is probably updating something (that same template?) to use a value pulled (possibly indirectly) from the cluster config the installer is pushing.
nhosoi
steven-ellis
markmc
zeenix
If your libvirt default storage pool is not /var/lib/libvirt/images then the libvirt-machine-controller fails to create the workers:
I worked around it with a bind mount. If it's not configurable then it would be handy if it was. (also, there's a typo "Coud" in the error message)