Add support for ceph storage using librbd

[shawniverson]

This would be great feature for kimchi.

http://docs.ceph.com/docs/jewel/rbd/libvirt/

[Azendale]

I would really like to see good Ceph support.

Good Ceph support in some kind of a Libvirt/KVM web GUI (Kimchi) is the last piece preventing me from being able to build a good Ceph+KVM cluster where I can maintain any hardware I need (save for the network switch) without downtime. (I can live migrate away from hypervisor nodes to work on them, and with Ceph is really easy in that respect).

I would be willing to pay $300 for good Ceph support. (Paypal, BitCoin, something like that). It needs to be able to do the following:

• Define the Ceph storage secret through the GUI.
• Be able to add Ceph RBD based disks to a VM through the GUI.
• Be able to add an RBD pool as a storage pool
• Be able to mount a CephFS filesystem (or subpath on that filesystem) as an ISO pool (I have tested mounting CephFS and then using the folder as an ISO pool and this does work)
• Being able to live migrate machines using RBD based disks
• Being able to live migrate machines with ISOs from a CephFS mount (without re-copying the ISO from the CephFS back into the same CephFS)

If you are interested, please contact me. I'm willing to negotiate, or pay some of the $300 for parts of this list being completed. (I'm going to also be trying to figure out how to add support myself, but since I've never contributed to this project before, there is going to be some learning curve involved, and I am a busy student, so it may not happen fast. I don't want to duplicate effort, so it is good to know if someone else is already tackling part of this list.) I should be notified if this issue is updated, but you can also contact me via e mail at erik[dot]b[dot]andersen[ a t ]gmail[dot]com.

[Azendale]

An update on this: I have a branch of Kimchi that seems to be able to successfully add a Ceph RBD storage pool (and define the secret needed for that).

https://github.com/Azendale/kimchi/tree/rbd1

I can't seem to add a disk to a VM yet though. Each way I try to do it currently doesn't work. I've tried:

• Adding an existing volume in the pool to an existing machine. This doesn't work because for some reason, I do not get a list of volumes in the pool when I select add an existing disk.
• Creating a new volume from the Guest edit page. I get the error "KCHVMSTOR0008E: Error while creating new storage device: This is not a valid RAW disk image." I'm using the raw disk type there -- I'm not sure if that is the correct choice or if it should be 'rbd'.
• Creating a new volume in the pool on the pool page. It wants me to upload an image, but says the format is not supported. (I never picked a format here, so I'm not sure what is going on).

If anyone wants to take a look at the branch and help test/debug whether there are bugs with using RBD pool (or if it is just a user(me)=ID10T error/an error in what I'm picking as far as formats), that would be awesome. I've made good progress so far but I think I'm starting to get stuck.

[bryanapperson]

Hi Azendale, I have a use case for this feature as well in the homelab I am building. I may pick up your branch and try to get this feature working. Have you made any further progress?

[Azendale]

bryanapperson, I didn't get much farther. Not that I'm not still interested in this, just got busy with other things.

I came to the conclusion that libvirt itself needs a patch to allow you to use disks from a Ceph pool with a machine. It seems like you can define a Ceph pool and everything, but can't actually use it. (And in the past, when I used Ceph and libvirt through other projects, I happened to always directly define the disk with all the needed XML under the machine itself.) So I know using Ceph directly without a libvirt pool works (when you manually set the libvirt XML or use some other libvirt tool than Kimchi). And it seems I got close (or possibly did even accomplish) defining a Ceph libvirt pool. But it this point that is not useful for actually attaching a disk to a machine. (At least that's my understanding).

Apparently, in the past, someone did submit a patch to the libvirt mailing list to fix that. They had a few things that needed cleanup, and it never got cleaned up and applied. Now that patch won't apply cleanly to the current libvirt. So that might be somewhere to start/look: taking the past patch, cleaning it up, and then updating the patch to match the current libvirt. See https://www.redhat.com/archives/libvirt-users/2015-March/msg00124.html.

It seems the another option would be to add support in Kimchi for directly defining a disk on a VM. That is the way I've used Ceph disks in the past; I know that works. If that was done, it would make sense to add support directly defining disks of other types as well, but that would probably be very easy, and would not be strictly necessary.

[Azendale]

@bryanapperson An update on this: I might be closer. I think I either got or am very close to getting the old patches working for libvirt. That would make it so that defining a libvirt pool that uses Ceph/RBD as a backend would actually be useful -- you could actually use it to store a domain's (virtual machine's) disk image.

See: https://github.com/Azendale/libvirt

It needs testing though: I still need to get it built and packaged into a .deb to test against my Ceph cluster, so I haven't been able to test the new version of libvirt.

But that could get this really, really close. If the libvirt patch works, and the experimental changes to Kimchi to allow defining a RBD/Ceph pool all line up, we might have a working solution.

[Azendale]

@bryanapperson, I have this built into an untested .deb for Ubuntu (based on the Bionic Beaver sources). Haven't tested yet, but it should be a lot easier to test now with a .deb. https://www.dropbox.com/sh/o2ch9kzh1li99qt/AAAX-Gyb4wZZ8tY1MCowyDKxa?dl=0