Linking issues after upgrading to 18.10

[jkozlowski]

Expected Behavior

I would like to be able to easily link all the required libraries to run an spdk-enabled application. Any help would be appreciated

Current Behavior

Since upgrading to 18.10 from 18.07.1, I have been getting many linking issues. My code worked fine with the previous version, and only required linking with spdk.

Possible Solution

I saw that there have been changes related to linking and shared libraries, but the README.md still leads me to believe that I should be able to just link in libspdk.so and get all the required libraries.

Steps to Reproduce

Please have a look at this PR: https://github.com/jkozlowski/starfish/pull/8/files and more specifically at the linking issues in CI: https://circleci.com/gh/jkozlowski/starfish/74?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link. Previously just linking with spdk sufficed, now we need to manually build dpdk, link to it separately, then we ended up needing a bunch of tricks to get things like librte_mempool_ring linked (There is great discussion here: https://github.com/jkozlowski/starfish/issues/6#issue-383854607).

I would love to know if there is something simple we are missing here? I appreciate the code is in Rust, but I am hoping there is something that stands out that we're missing.

Context (Environment including OS version, SPDK version, etc.)

SPDK version: 18.10 OS: Ubuntu

[peluse]

Adding @lhoswdev who may have some insight....

[lhoswdev]

Prior to changes in the build for producing SPDK shared libraries in preparation to developing a spec file for making SPDK rpm packages, the standalone shared library, libspdk.so, included "everything-but-the-kitchen-sink". One problem with that was it made the assumption that one wanted to employ the DPDK for all of the runtime environment (aka "env") functions which historically has been the default. However, the SPDK was extended a while ago to enable consumers of the SPDK to employ their own env in place of the DPDK if desired. To make that model consistent with the SPDK libraries, the 18.10 libspdk.so no longer contains the objects associated with the DPDK thereby enabling one to both link with the libspdk.so and at the same time supply their own env libraries.

First, ensure you run configure with the option, --with-shared, and then build the SPDK. As for your SPDK-based application, try changing your build from a simple link of just libspdk.so (-lspdk) to something like:

-lspdk -lspdk_env_dpdk -ldpdk

You may also need to specify the system libraries:

-lnuma -ldl -lrt -luuid

plus, of course, any additional libraries your application may require (e.g. -libverbs -lrdmacm) if using RDMA, etc.

[lhoswdev]

Following up on my previous post, the linking with the single libdpdk.so (i.e. -ldpdk) assumed that the dpdk and dpdk-devel packages had been installed and the SPDK's configure included the option --with-dpdk=/usr/share/dpdk/x86_64-default-linuxapp-gcc (assuming x86_64). With this configuration, the build of the SPDK does not rely on the DPDK submodule sources, but instead on pre-built DPDK binaries (and headers) installed from the DPDK packages.

[jkozlowski]

Hi @lhoswdev,

First of all thank you for a thorough explanation. I did not realise that spdk now has abstracted away dpdk, so as to be able to run without it, that's pretty cool.

The PR I linked already added -lspdk -lspdk_env_dpdk -ldpdk to the build and that I think passed the linking and build stages, but failed at runtime: the issue is that rust links with --as-needed by default (and we haven't found a way to disable that), meaning that code in drivers/mempool/ring/rte_mempool_ring.c which is not directly referenced is not linked into the executable.

I think the PR also tried doing --with-dpdk and buildling dpdk manually, but it seems like that didn't work for some reason.

I think steps from now are: 1) Try adding the systems libraries (I have low conviction that will help, but maybe?) 2) Try --with-dpdk=/usr/share/dpdk/x86_64-default-linuxapp-gcc, because the PR is doing ./configure --with-shared --with-dpdk=/usr/local after buildling dpdk as such: sudo make install T=uname -p-native-linuxapp-gcc \ CONFIG_RTE_BUILD_SHARED_LIB=y DESTDIR=/usr/local (I don't quite understand this incantation)

Question for you: Do you have a suggestion around the the rust compiler linking with --as-needed? To me it makes sense to do it by default, but it seems like either spdk or dpdk are relying on that not being the case? Or is this a red herring? It certainly doesn't feel like it, because @jkryl got everything to run just fine by prepending invocations with LD_PRELOAD=/usr/local/lib/librte_mempool_ring.so.1.1, but maybe we got something wrong?

[jkryl]

I think the overall approach makes sense. I find a bit counter intuitive that I need to explicitly install dpdk shared libraries into /usr/local even though that spdk build system "knows" I'm using dpdk bundled in the git submodule (no --with-dpdk option), but apparently it's also the way how it works for static libraries so from this POV it is consistent.

The rest of our problems is rather caused by rust compiler (as mentioned by @jkozlowski ) which does not allow us to explicitly specify --whole-archive linker flag, thus some dpdk libraries which should be part of the binary are missing then. IMHO we can only hope that rustc will become more flexible and allow us to tweak this behaviour one day.

[lhoswdev]

If your Linux distro offers the dpdk and dpdk-devel packages (and they're "suitably" recent), then you can build the SPDK and SPDK-applications without having to build the DPDK at all.

@jkozlowski , earlier versions of GNU's linker considered all specified libraries as required regardless of any symbol references in preceding objects and libs on the linker line. Later, the --as-needed became the default behavior. I'll leave it as an exercise to the reader to discover when that occurred :grin: One can employ the --no-as-needed to reverse the behavior but keep in mind that the argument, itself, is positionally-sensitive on the linker line, and that there can be performance hits by the resulting load of shared libraries when they really are not needed.

[jkozlowski]

Ah, you're saying it's the default in the system linker. Cool, I understand. @jkryl has found a way to reference the dpdk code in our library as a workaround, so I think we'll go with that for now, even thought it sounds pretty hacky.

I think we can close this one out, I really appreciate the time you took to help us out! We'll reopen or start a new issue if we find something else.