Rust bindings for SPDK.
git submodule update --init --recursive
sudo spdk/scripts/pkgdep.sh
nasm
is optional dependency so it must be installed explicitly (for ubuntu):
sudo apt-get install nasm
./build.sh
cd spdk && make install
sudo cp build/libspdk_fat.so /usr/local/lib/
build.sh is a helper script for building SPDK fat library. It is quite
rough and may not fit all use cases. Configuring & building SPDK without
using it is perfectly fine. The script just shows a possible way how to
do that. The only thing which matters is that at the end of build process
there must be a library named libspdk_fat.so
somewhere in standard
library path.
Creating a crate with bindings for spdk lib is challenging:
The purpose of this doc is to suggest a solution for the last problem mentioned above. As for the first three problems we assume that:
Though there are ways how to address first two problems in rust using “features”, it would require more work for the initial implementation and there is also maintenance burden as spdk libraries and headers get removed or added between SPDK releases. That said it would be a good extension of spdk sys crate later when it gets more mature.
We wish to have a spdk-sys crate which:
Best practices follow from articles listed in the links section and exemplar rust sys crates (libgit2-sys).
Switching between different options is done by using rust “features” or environment variables. The default should be set to the most popular way of using the library which depends on the platform and the library itself. Detecting if the lib is installed on a system is done almost exclusively using pkg-config.
SPDK can be built static or dynamic. It is a set of following libraries:
SPDK:
app_rpc bdev bdevio bdev_delay bdev_error bdev_gpt bdev_iscsi bdev_lvol bdev_malloc bdev_null bdev_nvme bdev_passthru bdev_id bdev_rpc bdev_split bdev_virtio blob blob_bdev blobfs conf copy copy_it env_dpdk event event_bdev event_copy event_iscsi event_nbd event_net event_nvmf event_scsi event_vhost event_vmd ftl ioat iscsi json jsonrpc log log_rpc lvol nbd net notify nvme nvmf rpc rte_vhost scsi sock sock_posix thread tce tce_rpc util vhost virtio vmd
DPDK:
bus_pci bus_vdev cmdline compressdev cryptodev eal ethdev hash krgs mbuf mempool mempool_bucket mempool_ring meter net pci ring vhost
ISA-L:
isal
There is no monolithic libspdk.so and libdpdk.so containing all libraries above. When building SPDK with --with-shared
configure option, DPDK libs are built as static. This is a bug in SPDK build system and can be easily fixed.
The most preferred solution would be to build SPDK object archives and link them statically to app. We bang a head against a wall when doing so. Object archives are linked with as-needed or no-whole-archive linker flag in rust and all libs which are not explicitly used by the app are omitted. Since there is no way how to change the default linker behaviour in rust, the only viable
workaround is to pretend that we use all of the libraries by referencing a symbol from each of them. The problem is that some of the libs have only private symbols and we can’t reference them anyhow without patching them. There is now way how to make that when using vanilla SPDK (from upstream). Unless the rust build system is enhanced to support -whole-archive
linker option (ticket https://github.com/rust-lang/rust/issues/56306), we have to use shared libraries.
Shared libraries suffer from the same problem as static libs as rust build system uses as-needed
linker flag. But there is an elegant workaround for that. If we create one big shared library out of all smaller ones then it will be surely referenced because there will be surely at least one call to the library - if there was none then it would not make sense to require the lib by the app in the first place - and the dynamic lib loader must load it all to the memory in one piece . The only problem is that we must create this fat library (as I will be calling it) ourselves. Good news is that it is super simple. We just need to take all object archives and combine them into a single shlib:
cc -shared -o libspdk_fat.so -Wl,--whole-archive *.a -Wl,--no-whole-archive
The result is a suboptimal solution because the shared library must be built and installed to the system prior to building the app and deployed to a target system along with the app. Detecting if libspdk fat lib is installed on the system is cumbersome as we can’t use pkg-config. SPDK in general is missing support for pkg-config ( https://trello.com/c/uBM2PR4c/19-generate-pkg-config-pc-file-during-make-install ). In spite of all drawbacks it is kinda standard rust solution for sys crates and works with upstream spdk with no changes required.
High level steps of how to use spdk sys crate:
Phase 1:
Phase 2:
Cargo.toml
.The way how the fat shlib is deployed along with the app to a target system is out of scope. In case of a docker image it can be copied from the system where the image is built to a docker image. On other systems it may be delivered in form of a package as it is usually done for other system libraries.
Some projects using SPDK in rust will need to use their own version of SPDK which differs from the upstream. There is probably no better way than to clone spdk-sys repository on github and override spdk git submodule in the repository so that it points to their own version of SPDK. In dependencies section of Cargo.toml file must be used a github URL of the cloned spdk-sys repo.
The sys crate should come with tests. It is out of scope to test each function provided by SPDK. At minimum calling a single function from spdk would be sufficient. Later when support for opt-in modules is added, there should be a similar test for each module.