tianon / docker-qemu

Dockerization of supported QEMU releases
https://qemu.org
133 stars 34 forks source link

Enable support for RBD #11

Closed tlex closed 3 years ago

tlex commented 3 years ago

This PR adds support for Ceph/RBD, by doing the following:

This enables a basic usage such as:

sudo docker run --rm -it \
  -v /etc/ceph/ceph.conf:/etc/ceph/ceph.conf:ro \
  -v /etc/ceph/ceph.client.qemu.keyring:/etc/ceph/ceph.client.qemu.keyring:ro \
  tianon/qemu:latest \
    qemu-img create -f raw rbd:rbd/desktop:id=qemu 100G

Also, it enables running a VM with RBD volume:

sudo docker run --rm -it \
  --device /dev/kvm \
  -v /etc/ceph/ceph.conf:/etc/ceph/ceph.conf:ro \
  -v /etc/ceph/ceph.client.qemu.keyring:/etc/ceph/ceph.client.qemu.keyring:ro \
  -v /docker/mini.iso:/tmp/mini.iso \
  tianon/qemu:latest \
        qemu-system-x86_64 \
          -enable-kvm \
          -smp 4 \
          -m 8192 \
          -boot order=d \
          -device virtio-scsi-pci \
          -drive format=rbd,file=rbd:rbd/desktop:id=qemu,id=drive1,cache=writeback,if=none \
          -device driver=scsi-hd,drive=drive1,discard_granularity=512 \
          -netdev user,hostname=desktop,hostfwd=tcp::22-:22,id=net \
          -device virtio-net-pci,netdev=net \
          -serial stdio \
          -vnc :0 \
          -no-user-config \
          -cpu host \
          -nodefaults \
          -vga std \
          -cdrom /tmp/mini.iso

Note: The librbd1 and librados2 packages need to be installed in the image in order to use RBD devices. For those needing that, I suggest a simple script that gets mounted in the image (with docker run -v):

#!/usr/bin/env bash
apt-get -qq update
DEBIAN_FRONTEND=noninteractive apt-get -y --no-install-recommends librbd1
rm -rf /var/lib/apt/lists/*

set -x
"${@}"

Of course, this script can be modified to install the latest images from https://download.ceph.com/ (see https://docs.ceph.com/docs/master/install/get-packages/#configure-repositories-manually)

tlex commented 3 years ago

I've rebased to the current master

tlex commented 3 years ago

I've also fixed the failing check - regenerated the Dockerfiles

tianon commented 3 years ago

Arg sorry, I wish I had checked this earlier -- the image size impact of this is pretty high. Testing 5.1, before is ~531MB and after is ~597MB (a pretty sizeable jump). When I build without explicitly marking librbd1, the image size is ~532MB, so maybe that's a reasonable compromise (so that we technically support rbd, but librbd1 has to be installed before it can be used).

Would that work for your "needs a newer version of librbd.so" use case too? In other words, is the ABI compatible enough that you could just install the newer version instead and have it "just work" without issue?

If we end up going this way, it probably warrants a note similar to:

...
        --enable-xen \
        --enable-xfsctl \
# rbd support is enabled, but "librbd1" is not included since it adds ~60MB and is version-sensitive (https://github.com/tianon/docker-qemu/pull/11#issuecomment-689816553)
        --enable-rbd \
#       --enable-vde \
    ; \
    make -j "$(nproc)"; \
...
tlex commented 3 years ago

Would that work for your "needs a newer version of librbd.so" use case too? In other words, is the ABI compatible enough that you could just install the newer version instead and have it "just work" without issue?

I need to test this some more, but for now it looks good - I could create an image and start a VM, after installing ceph octopus (15.2) in the container.