search

tikv / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
120 stars 93 forks source link

BUG: MultiBatchWrite crash while writeCallback::AllowWriteBatch returns false #369

Open mzygQAQ opened 2 months ago

mzygQAQ commented 2 months ago

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

works well

Actual behavior

crash in debug profile

Steps to reproduce the behavior

(gdb) bt
#0  0x00007ffff6f0d1f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff6f0e8e8 in abort () from /lib64/libc.so.6
#2  0x00007ffff6f06266 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6f06312 in __assert_fail () from /lib64/libc.so.6
#4  0x0000555559a774d3 in rocksdb::port::Mutex::AssertHeld() ()
#5  0x00005555596801b6 in rocksdb::InstrumentedMutex::AssertHeld() ()
#6  0x000055555968642c in rocksdb::DBImpl::WaitForPendingWrites() ()
#7  0x00005555596ed0da in rocksdb::DBImpl::MultiBatchWriteImpl(rocksdb::WriteOptions const&, std::vector<rocksdb::WriteBatch*, std::allocator<rocksdb::WriteBatch*> >&&, rocksdb::WriteCallback*, unsigned long*, unsigned long, unsigned long*) ()
#8  0x00005555596ee64e in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*) ()
#9  0x00005555596ecd2f in rocksdb::DBImpl::WriteWithCallback(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*) ()
#10 0x0000555559d6e3b6 in rocksdb::titandb::BlobGCJob::RewriteValidKeyToLSM() ()
#11 0x0000555559d6d56b in rocksdb::titandb::BlobGCJob::Finish() ()
#12 0x0000555559abfe59 in rocksdb::titandb::TitanDBImpl::BackgroundGC(rocksdb::LogBuffer*, unsigned int) ()
#13 0x0000555559abf5f3 in rocksdb::titandb::TitanDBImpl::BackgroundCallGC() ()
#14 0x0000555559abf4a2 in rocksdb::titandb::TitanDBImpl::BGWorkGC(void*) ()

WaitForPendingWrites requires mutex_.AssertHeld() but not, this only occurs in titan' gc job who will set AllowWriteBatching=false

mzygQAQ commented 2 months ago

@Connor1996 @tabokie @Little-Wallace

method MultiBatchWriteImpl is not in official facebook/rocksdb, maybe this is a special function used for tikv only?

Connor1996 commented 1 week ago

PTAL @v01dstar

v01dstar commented 1 week ago

@Connor1996 @tabokie @Little-Wallace

method MultiBatchWriteImpl is not in official facebook/rocksdb, maybe this is a special function used for tikv only?

That's correct. MultiBatchWrite is an optimization made by PingCAP in tikv's RocksDB fork. https://github.com/tikv/rocksdb/pull/387