Deadlock in mutator code

I put together a simple 1 billion row insert test to test out the performance 
of secondary indexes.  I did this by writing a script that launches the 
load_generator five times with five different seeds.  I launched this script on 
test08 and test09.  Most of the load_generator processes finished, but a few of 
them deadlocked.  To run this test:

1. copy ~doug/index_test to your home directory on both test08 and test09 (or 
test00 and test01).

2. clean the database

3. Launch Hypertable

4. Create the test table by running ~/index_test/create-table.hql through the 
command interpreter

5. launch the ~/index_test/load.sh script as follows on both test08 and test09 
(or test00 and test01):

nohup ./load.sh&

6. Wait until some of the processes hang.

Here's where the load_generator process is deadlocking:

Thread 31 (Thread 0x7fee9d5f7700 (LWP 29746)):
#0  0x00000036f020dfe4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000036f0209318 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00000036f02091e7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000059d05f in boost::unique_lock<boost::mutex>::lock() ()
#4  0x000000000066c2f1 in 
Hypertable::TableMutatorQueue::add(Hypertable::ApplicationHandler*) ()
#5  0x00000000006ae814 in 
Hypertable::TableMutatorAsyncDispatchHandler::handle(boost::intrusive_ptr<Hypert
able::Event>&) ()
#6  0x000000000070d52d in Hypertable::IOHandlerData::handle_message_body() ()
#7  0x000000000070da9c in Hypertable::IOHandlerData::handle_event(epoll_event*, 
long) ()
#8  0x000000000071de06 in Hypertable::ReactorRunner::operator()() ()
#9  0x00007feea73b8cf0 in thread_proxy () from 
/opt/hypertable/doug/current/lib/libboost_thread.so.1.44.0
#10 0x00000036f02077e1 in start_thread () from /lib64/libpthread.so.0
#11 0x00000036efee577d in clone () from /lib64/libc.so.6

[...]

Thread 1 (Thread 0x7feea60b07e0 (LWP 29709)):
#0  0x00000036f020b3cc in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00000000006f83ea in 
Hypertable::DispatchHandlerSynchronizer::wait_for_reply(boost::intrusive_ptr<Hyp
ertable::Event>&) ()
#2  0x0000000000642d44 in 
Hypertable::RangeServerClient::do_create_scanner(Hypertable::CommAddress 
const&, Hypertable::TableIdentifier const&, Hypertable::RangeSpec const&, 
Hypertable::ScanSpec const&, Hypertable::ScanBlock&, unsigned int) ()
#3  0x000000000063e166 in 
Hypertable::RangeLocator::find(Hypertable::TableIdentifier const*, char const*, 
Hypertable::RangeLocationInfo*, Hypertable::Timer&, bool) ()
#4  0x000000000063ef9b in 
Hypertable::RangeLocator::find_loop(Hypertable::TableIdentifier const*, char 
const*, Hypertable::RangeLocationInfo*, Hypertable::Timer&, bool) ()
#5  0x000000000067b5b0 in 
Hypertable::TableMutatorAsyncScatterBuffer::set(Hypertable::SerializedKey, 
Hypertable::ByteString, unsigned long) ()
#6  0x000000000067bfbf in 
Hypertable::TableMutatorAsyncScatterBuffer::create_redo_buffer(unsigned int) ()
#7  0x00000000006758b8 in Hypertable::TableMutatorAsync::buffer_finish(unsigned 
int, int, bool) ()
#8  0x000000000067d6e2 in Hypertable::TableMutatorAsyncScatterBuffer::finish() 
()
#9  0x00000000006b0036 in Hypertable::TableMutatorAsyncHandler::run() ()
#10 0x0000000000669a0b in 
Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsyn
c*) ()
#11 0x00000000006714c5 in 
Hypertable::TableMutatorAsync::flush_with_tablequeue(Hypertable::TableMutator*, 
bool) ()
#12 0x00000000006735e8 in 
Hypertable::TableMutatorAsync::set_cells(__gnu_cxx::__normal_iterator<Hypertable
::Cell const*, std::vector<Hypertable::Cell, 
Hypertable::PageArenaAllocator<Hypertable::Cell, Hypertable::PageArena<char, 
Hypertable::DefaultPageAllocator> > > >, 
__gnu_cxx::__normal_iterator<Hypertable::Cell const*, 
std::vector<Hypertable::Cell, Hypertable::PageArenaAllocator<Hypertable::Cell, 
Hypertable::PageArena<char, Hypertable::DefaultPageAllocator> > > >) ()
#13 0x000000000066a1bf in 
Hypertable::TableMutator::set_cells(__gnu_cxx::__normal_iterator<Hypertable::Cel
l const*, std::vector<Hypertable::Cell, 
Hypertable::PageArenaAllocator<Hypertable::Cell, Hypertable::PageArena<char, 
Hypertable::DefaultPageAllocator> > > >, 
__gnu_cxx::__normal_iterator<Hypertable::Cell const*, 
std::vector<Hypertable::Cell, Hypertable::PageArenaAllocator<Hypertable::Cell, 
Hypertable::PageArena<char, Hypertable::DefaultPageAllocator> > > >) ()
#14 0x000000000058f6e4 in LoadClient::set_cells(std::vector<Hypertable::Cell, 
Hypertable::PageArenaAllocator<Hypertable::Cell, Hypertable::PageArena<char, 
Hypertable::DefaultPageAllocator> > > const&) ()
#15 0x000000000057dbdb in 
generate_update_load(boost::intrusive_ptr<Hypertable::Properties>&, 
std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, 
bool, unsigned long, bool, std::basic_string<char, std::char_traits<char>, 
std::allocator<char> >&, int, bool) ()
#16 0x000000000058272a in main ()

You'll need to use the latest v0.9.5 branch
Original issue reported on code.google.com by nuggetwh...@gmail.com on 20 Mar 2012 at 10:59
linearregression / hypertable

Deadlock in mutator code #809
