ceph / ceph-rust

Rust-lang interface to Ceph.
Apache License 2.0
98 stars 43 forks source link

Add support for Rados striper #46

Closed bancek closed 5 years ago

bancek commented 5 years ago

This PR adds support for Rados striper (libradosstriper). It is disabled by default and can be enabled with feature rados_striper.

The PR also adds initial Docker support for CI. For now it can be run manually with make test-docker. Running with Travis CI still needs to be integrated. Some scripts were copied (and slightly modified to work with with Nautilus) from https://github.com/ceph/go-ceph

Closes #45

cholcombe973 commented 5 years ago

Wow this is excellent work! Have you gone over the code with valgrind yet? I want to say it looks correct to me but calling c can be tricky sometimes.

bancek commented 5 years ago

We've used read, write, remove and truncate sync APIs in one of our services so these work. I haven't tested the others, but the signatures are identical to the IOContext ones, so I've just copied and renamed the code.

Do you have any documentation for testing with Valgrind?

cholcombe973 commented 5 years ago

Yeah I figured you copied the IOContext examples. The code looks very similar :). If you don't want to valgrind the code i can just do it later when I have some free time. Mostly what i've done in the past is compile one of the examples in debug and launch it with valgrind. It'll run slowly and then ensure all the memory manipulation is correct. So it would be something like cargo build --debug example/rados_striper valgrind ./target/debug/rados_striper and then see what it produces for output.

bancek commented 5 years ago

Valgrind 3.11 (latest in Ubuntu 16.04) fails with the following error:

vex amd64->IR: unhandled instruction bytes: 0xF 0xC7 0xF0 0x89 0x6 0xF 0x42 0xC1
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==695== valgrind: Unrecognised instruction at address 0x69b0bd5.
==695==    at 0x69B0BD5: ??? (in /usr/lib/ceph/libceph-common.so.0)
==695==    by 0x69B0D71: std::random_device::_M_getval() (in /usr/lib/ceph/libceph-common.so.0)

This seems to be a bug in Valgrind: https://bugs.kde.org/show_bug.cgi?id=353370

I've tried with the latest Valgrind:

make shell-docker
apt-get install libc6-dbg
wget -O /tmp/valgrind_3.15.0-1ubuntu3_amd64.deb http://launchpadlibrarian.net/440389681/valgrind_3.15.0-1ubuntu3_amd64.deb
dpkg -i /tmp/valgrind_3.15.0-1ubuntu3_amd64.deb

. /setup-micro-osd.sh
cargo build --features rados_striper --example rados_striper
valgrind target/debug/examples/rados_striper
root@dffef8c4be1b:/ceph-rust# valgrind target/debug/examples/rados_striper
==443== Memcheck, a memory error detector
==443== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==443== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==443== Command: target/debug/examples/rados_striper
==443==
Connecting to ceph
Creating pool ceph-rust-test
Creating ioctx
Creating rados striper
Writing test object
Creating ioctx
Creating rados striper
Getting test object stat
Reading test object
Deleting pool ceph-rust-test
==443==
==443== HEAP SUMMARY:
==443==     in use at exit: 10,156 bytes in 77 blocks
==443==   total heap usage: 36,985 allocs, 36,908 frees, 16,269,989 bytes allocated
==443==
==443== LEAK SUMMARY:
==443==    definitely lost: 0 bytes in 0 blocks
==443==    indirectly lost: 0 bytes in 0 blocks
==443==      possibly lost: 0 bytes in 0 blocks
==443==    still reachable: 10,156 bytes in 77 blocks
==443==         suppressed: 0 bytes in 0 blocks
==443== Rerun with --leak-check=full to see details of leaked memory
==443==
==443== For lists of detected and suppressed errors, rerun with: -s
==443== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
cholcombe973 commented 5 years ago

Ah yeah that's perfect. Nice work!