CCI-MOC / lsvd-rbd

Log Structured Virtual Disk
GNU General Public License v2.0
8 stars 5 forks source link

Implement RPC support for the LSVD bdev backend #40

Closed knikolla closed 2 months ago

knikolla commented 3 months ago

Currently, the lsvd binary starts up SPDK, registers a LSVD bdev backend and a single bdev, and creates an NVMe over fabrics TCP transport. It is not possible to create others because there is no support for issuing RPC commands to register them.

From an email from Isaac

We were talking about the canonical way to use LSVD going forward, and it seems like the best way forward is to integrate LSVD into the rpc framework.

The implementation shouldn’t be difficult, just tedious. Take a look at https://github.com/spdk/spdk/blob/master/module/bdev/rbd/bdev_rbd_rpc.c for reference on how it’s supposed to be done. The equivalent of bdev_rbd.c is for lsvd is https://github.com/CCI-MOC/lsvd-rbd/blob/main/src/bdev_lsvd.cc.

The main idea here is that the file itself defines SPDK_RPC_REGISTER sections, which defines a new rpc endpoint. At startup SPDK will scan through the binary and libraries for defined RPC endpoints and do the actual registration.The registration takes a function pointer for the entry point, and since the commands are passed in as json it’s probably easiest to just use their json decoding framework (if you want to pull in a cpp json framework that’s nicer feel free).

The main thing to note is that the RBD bdev rpc implementation is very async-driven; we can avoid the c-style callbackfn hell to a certain extent because the LSVD api is designed with some of this in mind. Just keep lifetimes in mind.

LSVD currently creates its own RBD cluster on image creation if one is not passed in iirc, so we don’t need to implement or integrate any of the RBD cluster functions. The only endpoints we need to register should be the bdev_lsvd_create and delete; everything else should be fine being left out of the implementation. We can always add them later if the need arises.

  • Isaac Khor
knikolla commented 2 months ago

Closed by #41