search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
28.65k stars 6.33k forks source link

Core dump while using rocksdbjni-6.11.4 #7730

Closed gaojieliu closed 3 years ago

gaojieliu commented 3 years ago

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

Actual behavior

Steps to reproduce the behavior

Stacktrace:

(gdb) bt
#0  0x00007fa34c46f2c7 in raise () from /lib64/libc.so.6
#1  0x00007fa34c4709b8 in abort () from /lib64/libc.so.6
#2  0x00007fa34b150b35 in os::abort(bool) () from /export/apps/jdk/JDK-1_8_0_172-zulu/jre/lib/amd64/server/libjvm.so
#3  0x00007fa34b2e2a13 in VMError::report_and_die() () from /export/apps/jdk/JDK-1_8_0_172-zulu/jre/lib/amd64/server/libjvm.so
#4  0x00007fa34b1573a2 in JVM_handle_linux_signal () from /export/apps/jdk/JDK-1_8_0_172-zulu/jre/lib/amd64/server/libjvm.so
#5  0x00007fa34b14cf03 in signalHandler(int, siginfo*, void*) () from /export/apps/jdk/JDK-1_8_0_172-zulu/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x00007fa30aa9976b in rocksdb::ThreadLocalPtr::CompareAndSwap(void*, void*&) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#8  0x00007fa30a7ccd63 in rocksdb::ColumnFamilyData::ReturnThreadLocalSuperVersion(rocksdb::SuperVersion*) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#9  0x00007fa30a80cc48 in rocksdb::DBImpl::ReturnAndCleanupSuperVersion(rocksdb::ColumnFamilyData*, rocksdb::SuperVersion*) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#10 0x00007fa30a80d111 in rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#11 0x00007fa30a80d8e7 in rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, std::string*) () from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#12 0x00007fa30a804fb7 in rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) () from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#13 0x00007fa30a76c0f6 in rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#14 0x00007fa30a76804d in rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#15 0x00007fa30a75fc5f in rocksdb_get_helper(JNIEnv_*, rocksdb::DB*, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int, _jbyteArray*, int, int, bool*) ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#16 0x00007fa30a75fd63 in Java_org_rocksdb_RocksDB_get__J_3BII_3BII ()
   from /export/content/lid/apps/venice-server/03c93b0c52aa4c21ffdbe8f960cb011f25e45906/tmp/librocksdbjni3760533038315774975.so
#17 0x00002aaaae184352 in ?? ()

We didn't find out a good way to reproduce this issue reliably, and we would like to check with RocksDB dev team to see whether this stacktrace is familiar or not.

adamretter commented 3 years ago

@pdillinger @ajkr Have you seen anything like this originating from rocksdb::ColumnFamilyData::ReturnThreadLocalSuperVersion?

gaojieliu commented 3 years ago

@pdillinger @ajkr Kindly ping again. Have you seen anything similar as this? Do you know whether the latest release has already fixed this issue or not?

pdillinger commented 3 years ago

Internal search doesn't show anything.

DeepWu commented 3 years ago

Hello, I have encountered a similar mistake, if you have any ideas, I will be very grateful @adamretter

(gdb) where

0 0x00007f7fe867e207 in raise () from /lib64/libc.so.6

1 0x00007f7fe867f8f8 in abort () from /lib64/libc.so.6

2 0x00007f7fe7d99169 in os::abort(bool, void, void const) () from /opt/bin/etl-jre-x86_64/lib/server/libjvm.so

3 0x00007f7fe7fc02fa in VMError::report_and_die(int, char const, char const, __va_list_tag, Thread, unsigned char, void, void, char const, int, unsigned long) ()

from /opt/etl-jre-x86_64/lib/server/libjvm.so

4 0x00007f7fe7fc0b7b in VMError::report_and_die(Thread, unsigned int, unsigned char, void, void, char const*, ...) ()

from /opt/bin/etl-jre-x86_64/lib/server/libjvm.so

5 0x00007f7fe7fc0bae in VMError::report_and_die(Thread, unsigned int, unsigned char, void, void) ()

from /opt/etl-jre-x86_64/lib/server/libjvm.so

6 0x00007f7fe7da3a10 in JVM_handle_linux_signal () from /opt/etl-jre-x86_64/lib/server/libjvm.so

7 0x00007f7fe7d97c08 in signalHandler(int, siginfo, void) () from /opt/etl-jre-x86_64/lib/server/libjvm.so

8 《 signal handler called》

9 0x00007f7f53910b7f in rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions) () from /tmp/librocksdbjni5352942002711263756.so

10 0x00007f7f5391162d in rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle, rocksdb::Slice const&, rocksdb::PinnableSlice) () from /tmp/librocksdbjni5352942002711263756.so

11 0x00007f7f53876b46 in rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle, rocksdb::Slice const&, std::string) () from /tmp/librocksdbjni5352942002711263756.so

12 0x00007f7f5387466d in rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*) () from /tmp/librocksdbjni5352942002711263756.so

13 0x00007f7f5386f972 in rocksdb_gethelper(JNIEnv, rocksdb::DB, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle, _jbyteArray, int, int) () from /tmp/librocksdbjni5352942002711263756.so

14 0x00007f7f5386fa69 in Java_org_rocksdb_RocksDB_get__J_3BII () from /tmp/librocksdbjni5352942002711263756.so

15 0x00007f7fd0aa9321 in ?? ()

16 0x00000007002e20b8 in ?? ()

ltamasi commented 3 years ago

This could be another manifestation of the issue fixed in #8605 .

@gaojieliu Could you check if you still see this issue after that fix?

ltamasi commented 3 years ago

Assuming fixed, please feel free to reopen if you still see this after the fix mentioned above.