search

serokell / rocksdb-haskell

Haskell bindings to RocksDB (http://rocksdb.org)
BSD 3-Clause "New" or "Revised" License
27 stars 10 forks source link

Thread-safety #9

Open domenkozar opened 6 years ago

domenkozar commented 6 years ago

If one thread closes DB and another one tries to access it, you get a segfault:

(gdb) thread apply all bt

Thread 8 (Thread 0x7f91670cf9c0 (LWP 15587)):
#0  0x00007f9164b9ddaf in fallocate64 () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6
#1  0x00007f9166818043 in rocksdb::PosixWritableFile::Close() ()
   from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#2  0x00007f9166812426 in rocksdb::WritableFileWriter::Close() ()
   from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#3  0x00007f916674409c in rocksdb::log::Writer::~Writer() ()
   from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#4  0x00007f916677404b in rocksdb::VersionSet::~VersionSet() ()
   from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#5  0x00007f91666f6957 in rocksdb::DBImpl::~DBImpl() () from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#6  0x00007f91666f6f81 in rocksdb::DBImpl::~DBImpl() () from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#7  0x00007f9166698485 in rocksdb_close () from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#8  0x0000000000ba05c6 in ?? ()
#9  0x00000042003f7d28 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f91555e8700 (LWP 27599)):
#0  0x00007f916508036f in pthread_cond_wait@@GLIBC_2.3.2 () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libpthread.so.0
#1  0x00007f9166848c54 in rocksdb::ThreadPoolImpl::BGThread(unsigned long) ()
   from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#2  0x00007f9166848e53 in rocksdb::BGThreadWrapper(void*) ()
   from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#3  0x00007f916507a234 in start_thread () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libpthread.so.0
#4  0x00007f9164ba675f in clone () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6

Thread 1 (Thread 0x7f9157fff700 (LWP 15592)):
#0  0x00007f9166698c7e in rocksdb_get () from /nix/store/2svz1v57kgnlhks2j40fzf0vrqrnpa0g-rocksdb-5.1.2/lib/librocksdb.so.5.1
#1  0x0000000000ba8ec3 in ?? ()
#2  0x0000004202c20e50 in ?? ()
#3  0x0000000000000000 in ?? ()
domenkozar commented 6 years ago

Way to reproduce:

[root@staging:~]# git clone https://github.com/input-output-hk/iohk-ops
[root@staging:~]# cd iohk-ops/
[root@staging:~]# git checkout slow-fallocate
[root@staging:~]# cd slow-fallocate
[root@staging:~]# nix-build
# confirm fallocate is slower
[root@staging:~]# time LD_PRELOAD=./result/lib/libslow-fallocate.so fallocate img.bin -l 1g
# confirm the current cardano args
[root@staging:~]# ps aux | grep cardano
[root@staging:~]# systemctl stop cardano-node
[root@staging:~]# sudo -u cardano-node /bin/sh
# run cardano with the normal args, and the LD_PRELOAD set
LD_PRELOAD=/nix/store/i064x9kmiy9b143li7i4621cbsqdd351-slow-fallocate/lib/libslow-fallocate.so /nix/store/wn9vva936qpbwp46rs0bv0jrpd1iq97z-cardano-sl-0.6.2/bin/cardano-node-simple .....
# ctrl+c to stop after a while
[node:DEBUG] [2017-09-25 15:43:02 UTC] waiting for [] outbound inbound
[node:ERROR] [2017-09-25 15:43:02 UTC] exception while stopping node thread killed
[node:ERROR] [2017-09-25 15:43:02 UTC] exception while stopping node thread killed
fallocate(22, 1, 0, 73819750) == 0
fallocate(22, 3, 3809, 73815941) == 0
fallocate(16, 3, 137, 4613597) == 0
fallocate(15, 3, 548, 4193756) == 0
Segmentation fault (core dumped)
[root@staging:~]# systemctl start cardano-node

Proposed API change: https://gist.github.com/cleverca22/77582806a7eb8c5bb34655af16268295