search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
28.46k stars 6.3k forks source link

db_bench readrandom value_size 1024B stuck at 72m ops #4802

Open HustCoderHu opened 5 years ago

HustCoderHu commented 5 years ago

Expected behavior

Would have finished at least.

Actual behavior

It got stuck at 72300000 ops. Ram and ssd still had much free space(ram 10g+, ssd 500g+), and cpu utilization was nearly 0. I guess deadlock occurred

Steps to reproduce the behavior

version: last commit on Dec 15, 2018 a914a1c6dc37c63ac06186767235aa9d8f17469c

cmake -DCMAKE_BUILD_TYPE=Release -DWITH_SNAPPY=1 -DUSE_RTTI=1 ..
make db_bench
../Release/db_bench \
--benchmarks="fillrandom,readrandom" \
--db="/mnt/ssd" \
--key_size=16 \
--reads=100000 \
--write_buffer_size=`expr 64 \* 1048576` \
--max_bytes_for_level_base=`expr 256 \* 1048576` \
--use_existing_db=0 \
--max_background_compactions=16 \
--value_size=1024 \
--num=80000000

heres the LOG https://raw.githubusercontent.com/HustCoderHu/CodingPractice/master/database/LOG

miasantreble commented 5 years ago

@HustCoderHu thanks for reporting the issue, would it be possible to get pstack on the process when the suspected deadlock happens? that would help us identify the problem.

HustCoderHu commented 5 years ago 
Thread 21 (Thread 0x7f514b7fe700 (LWP 84397)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51acf4eb3d in rocksdb::port::CondVar::Wait() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#2  0x00007f51acd04b38 in rocksdb::InstrumentedCondVar::Wait() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51acdd9899 in rocksdb::Benchmark::DoWrite(rocksdb::ThreadState*, rocksdb::Benchmark::WriteMode) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51acdbdd74 in rocksdb::Benchmark::ThreadBody(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#5  0x00007f51acf4efa2 in rocksdb::(anonymous namespace)::StartThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#6  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f514affd700 (LWP 84396)):
#0  0x00007f51ac3751e6 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51acf4eba2 in rocksdb::port::CondVar::TimedWait(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#2  0x00007f51acbc34ab in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#4  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f518dffb700 (LWP 84380)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51acf4eb3d in rocksdb::port::CondVar::Wait() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#2  0x00007f51acd04b38 in rocksdb::InstrumentedCondVar::Wait() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace2b688 in rocksdb::DeleteScheduler::BackgroundEmptyTrash() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f518e7fc700 (LWP 84361)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f518effd700 (LWP 84360)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f51907f8700 (LWP 84359)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f5190ff9700 (LWP 84358)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f51917fa700 (LWP 84357)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f5191ffb700 (LWP 84356)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f51927fc700 (LWP 84355)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f5192ffd700 (LWP 84354)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f51937fe700 (LWP 84353)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f518f7fe700 (LWP 84352)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f5193fff700 (LWP 84351)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f51a8e88700 (LWP 84350)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f51a9689700 (LWP 84349)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f51a9e8a700 (LWP 84348)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f51aa68b700 (LWP 84347)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f51aae8c700 (LWP 84345)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f51ab68d700 (LWP 84344)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51ac09523c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#2  0x00007f51ace4b679 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51ace4ba0f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51ac09b53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f51ac36e609 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f51ab7a7e6f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f51ad4514c0 (LWP 84340)):
#0  0x00007f51ac374c3b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f51acf4eb3d in rocksdb::port::CondVar::Wait() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#2  0x00007f51acdce758 in rocksdb::Benchmark::RunBenchmark(int, rocksdb::Slice, void (rocksdb::Benchmark::*)(rocksdb::ThreadState*)) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#3  0x00007f51acdd3919 in rocksdb::Benchmark::Run() () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#4  0x00007f51acdb707f in rocksdb::db_bench_tool(int, char**) () from /home/hustzyw/rocksdb/rocksdb/Release/librocksdb.so.5
#5  0x00007f51ab6af03a in __libc_start_main () from /lib64/libc.so.6
#6  0x000000000041395a in _start ()
miasantreble commented 5 years ago

I ran the command and it completed successfully:


TEST_TMPDIR=/tmp ./db_bench --benchmarks="fillrandom,readrandom" --key_size=16 --reads=100000 --write_buffer_size=`expr 64 \* 1048576` --max_bytes_for_level_base=`expr 256 \* 1048576` --use_existing_db=0 --max_background_compactions=16 --value_size=1024 --num=80000000
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 5.18
Date:       Thu Dec 20 11:07:58 2018
CPU:        24 * Intel Core Processor (Skylake)
CPUCache:   16384 KB
Keys:       16 bytes each
Values:     1024 bytes each (512 bytes after compression)
Entries:    80000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    79345.7 MB (estimated)
FileSize:   40283.2 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [/tmp/dbbench]
fillrandom   :      15.092 micros/op 66262 ops/sec;   65.7 MB/s
DB path: [/tmp/dbbench]
readrandom   :    1006.608 micros/op 993 ops/sec;    1.0 MB/s (100000 of 100000 found)
mdcallag commented 5 years ago

Given the command line and the thread stack show DoWrite I assume that this occurs with fillrandom, not readrandom.

Grep LOG for "error" and first is this which is fixed by using ulimit to change per-user limit for open SST or by configuring RocksDB to keep fewer open files.

2018/12/19-17:04:13.811622 7f51917fa700 [WARN] [db/db_impl_compaction_flush.cc:2601] Compaction error: IO error: While open a file for random read: /mnt/ssd//005648.sst: Too many open files