To anyone seeing this: THE REAL WORKING README HERE

Before you proceed, here's some facts about this project...

Maintenance

All previous issues get ZERO REPLY from authors, dating all the way back to 2019. Last updated many years ago.

Instructions

The README is VERY OUTDATED and WRONG. It instructs you to install bmi for mercury, which is deprecated by mercury. And the run command? It simply won't work out of the box due to bugs in code (more on that later). Moreover, even if you do ever get to start the server, the client can never connect to it. There is ZERO document about proper configuration and usage.

Bugs in code

I have been working on this code for several days and found some serious bugs. You can check it yourself.

On the server side, the default config for EnvName is posix, and the README says rados or sth else, but in its parsing process it only works with default (or perhaps unbufferedio, I didn't try that) and rados is not even implemented.
The code is defaulted to use bmi+tcp out of the box and the README keep telling you to use mercury rpc, but the RPCOptions's rpc type is hardcoded to be kSocketRPC regardless of whether PDLFS_MERCURY_RPC is defined or not, meaning it NEVER USES MERCURY. The ctest about rpc passes because the test code is hardcoded to use tcp and udp separately.

Functionality

FUNDAMENTAL FS semantics like rmdir and rename are missing, so mdtest won't even run. And they even call this a filesystem lol.

What I want to say is

If you have a choice, just get out and don't use this. This won't deliver what was claimed in the paper. Objections are welcomed, but please show me the real code ;).

How to get it work (in Ubuntu-22.04)

Install libfabric.

Don't install it with apt in Ubuntu 22.04, that one's version is too old. Get a new version and compile it yourself.

wget https://github.com/ofiwg/libfabric/releases/download/v1.21.0/libfabric-1.21.0.tar.bz2
tar -xjf libfabric-1.21.0.tar.bz2
cd libfabric-1.21.0
./configure && make -j
sudo checkinstall
# or you can do `sudo make install` if you don't want to `apt install checkinstall`

Install mercury

# extra dependency
sudo apt install libjson-c-dev
# idk why, but mercury says it needs this.
sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope"
git clone --recurse-submodules https://github.com/mercury-hpc/mercury.git
cd mercury
mkdir build && cd build
ccmake ..
# remember to configure MERCURY_USE_OFI to ON
make -j
sudo checkinstall -pkgname mercury

Fix the bugs in code

penglb3@node1:~/deltafs$ git diff
diff --git a/external/pdlfs-common/src/mercury/mercury_test.cc b/external/pdlfs-common/src/mercury/mercury_test.cc
index 707a3f1..d9c3cc5 100644
--- a/external/pdlfs-common/src/mercury/mercury_test.cc
+++ b/external/pdlfs-common/src/mercury/mercury_test.cc
@@ -17,7 +17,7 @@
 namespace pdlfs {
 namespace rpc {

-static const std::string kProto = "bmi+tcp://127.0.0.1:10101";
+static const std::string kProto = "ofi+tcp://127.0.0.1:10101";
 // True if multiple client threads can call RPC simultaneously
 static const bool kAllowConcurrentRPC = false;
diff --git a/external/pdlfs-common/src/rpc.cc b/external/pdlfs-common/src/rpc.cc
index 6558daf..188c0e0 100644
--- a/external/pdlfs-common/src/rpc.cc
+++ b/external/pdlfs-common/src/rpc.cc
@@ -30,7 +30,11 @@
 namespace pdlfs {

 RPCOptions::RPCOptions()
+#ifdef PDLFS_MERCURY_RPC
+    : impl(rpc::kMercuryRPC),
+#else
     : impl(rpc::kSocketRPC),
+#endif
       mode(rpc::kServerClient),
       rpc_timeout(5000000),
       num_rpc_threads(1),
diff --git a/src/libdeltafs/deltafs_conf.cc b/src/libdeltafs/deltafs_conf.cc
index b9a0480..dbee1fe 100644
--- a/src/libdeltafs/deltafs_conf.cc
+++ b/src/libdeltafs/deltafs_conf.cc
@@ -49,7 +49,7 @@ namespace config {
 DEFINE_FLAG(NumOfMetadataSrvs, "1")
 DEFINE_FLAG(NumOfVirMetadataSrvs, "1")
 DEFINE_FLAG(InstanceId, "0")
-DEFINE_FLAG(RPCProto, "bmi+tcp")
+DEFINE_FLAG(RPCProto, "ofi+tcp")
 DEFINE_FLAG(MDSTracing, "false")
 DEFINE_FLAG(MetadataSrvAddrs, "")
 DEFINE_FLAG(MaxNumOfOpenFiles, "1000")
@@ -66,7 +66,7 @@ DEFINE_FLAG(VerifyChecksums, "false")
 DEFINE_FLAG(Inputs, "/tmp/deltafs_inputs")
 DEFINE_FLAG(Outputs, "/tmp/deltafs_outputs")
 DEFINE_FLAG(RunDir, "/tmp/deltafs_run")
-DEFINE_FLAG(EnvName, "posix")
+DEFINE_FLAG(EnvName, "default")
 DEFINE_FLAG(EnvConf, "")
 DEFINE_FLAG(FioName, "posix")
 DEFINE_FLAG(FioConf, "")

Then compile the deltafs code. Enable mercury, MPI. To run the server:

DELTAFS_MetadataSrvAddrs=127.0.0.1:10101 ./build/src/server/deltafs-srvr -v=1 -logtostderr

Run the client shell

DELTAFS_MetadataSrvAddrs=127.0.0.1:10101 DELTAFS_NumOfMetadataSrvs=1 ./build/src/cmds/deltafs-shell -v=1 -logtostderr

pdlfs / deltafs