HSET Failure

[ipapapa]

ARDB seems to be dying when HSET Is done. A few more inforrmation can be found in the following core dump:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/apps/ardb/bin/ardb-server /apps/ardb/conf/rocksdb.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:799
799 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory.
[Current thread is 1 (Thread 0x7f64929f5700 (LWP 5052))]
(gdb) 
(gdb) bt
#0  __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:799
#1  0x00000000004d3d29 in ardb::Data::Compare (this=this@entry=0x7f6491f08d60, right=..., alpha_cmp=alpha_cmp@entry=false) at types.cpp:334
#2  0x000000000059b609 in ardb::Data::operator< (other=..., this=<optimized out>) at ./types.hpp:204
#3  ardb::KeyPrefix::operator< (other=..., this=<optimized out>) at ./context.hpp:85
#4  std::less<ardb::KeyPrefix>::operator() (this=<optimized out>, __y=..., __x=...) at /usr/include/c++/5/bits/stl_function.h:387
#5  btree::btree_key_comparer<ardb::KeyPrefix, btree::btree_key_compare_to_adapter<std::less<ardb::KeyPrefix> >, false>::bool_compare (comp=..., y=..., x=...) at /src/src/../deps/cpp-btree/btree.h:236
#6  btree::btree_compare_keys<ardb::KeyPrefix, btree::btree_key_compare_to_adapter<std::less<ardb::KeyPrefix> > > (comp=..., y=..., x=...) at /src/src/../deps/cpp-btree/btree.h:269
#7  btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >::binary_search_plain_compare<btree::btree_key_compare_to_adapter<std::less<ardb::KeyPrefix> > > (comp=..., e=1, s=0, k=..., this=<optimized out>) at /src/src/../deps/cpp-btree/btree.h:638
#8  btree::btree_binary_search_plain_compare<ardb::KeyPrefix, btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >, btree::btree_key_compare_to_adapter<std::less<ardb::KeyPrefix> > >::lower_bound (comp=..., n=..., k=...) at /src/src/../deps/cpp-btree/btree.h:404
#9  btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >::lower_bound<btree::btree_key_compare_to_adapter<std::less<ardb::KeyPrefix> > > (comp=..., k=..., this=<optimized out>) at /src/src/../deps/cpp-btree/btree.h:592
#10 btree::btree<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >::internal_locate_plain_compare<btree::btree_iterator<btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>&, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>*> > (this=<optimized out>, iter=..., key=...) at /src/src/../deps/cpp-btree/btree.h:2247
#11 btree::btree_internal_locate_plain_compare::dispatch<ardb::KeyPrefix, btree::btree<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >, btree::btree_iterator<btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>&, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>*> > (t=..., iter=..., k=...) at /src/src/../deps/cpp-btree/btree.h:850
#12 btree::btree<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >::internal_locate<btree::btree_iterator<btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>&, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>*> > (this=<optimized out>, iter=..., key=...) at /src/src/../deps/cpp-btree/btree.h:2240
#13 btree::btree<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >::internal_find_unique<btree::btree_iterator<btree::btree_node<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>&, std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*>*> > (iter=..., key=..., this=0x7ffefd637830) at /src/src/../deps/cpp-btree/btree.h:2310
#14 btree::btree<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> >::find_unique (key=..., this=0x7ffefd637830)
    at /src/src/../deps/cpp-btree/btree.h:1076
#15 btree::btree_unique_container<btree::btree<btree::btree_map_params<ardb::KeyPrefix, ardb::ThreadMutexLock*, std::less<ardb::KeyPrefix>, std::allocator<std::pair<ardb::KeyPrefix const, ardb::ThreadMutexLock*> >, 256> > >::find (key=..., 
    this=0x7ffefd637830) at /src/src/../deps/cpp-btree/btree_container.h:183
#16 ardb::Ardb::UnlockKey (this=0x7ffefd637310, lk=...) at db/db.cpp:790
#17 0x000000000059b8d5 in ardb::Ardb::KeyLockGuard::~KeyLockGuard (this=0x7f64929f2e00, __in_chrg=<optimized out>) at db/db.cpp:122
#18 0x000000000058f3a5 in ardb::Ardb::HSet (this=0x7ffefd637310, ctx=..., cmd=...) at command/t_hash.cpp:188
#19 0x000000000059504a in ardb::Ardb::DoCall (this=this@entry=0x7ffefd637310, ctx=..., setting=..., args=...) at db/db.cpp:1377
#20 0x000000000059a997 in ardb::Ardb::Call (this=0x7ffefd637310, ctx=..., args=...) at db/db.cpp:1545
#21 0x00000000004cd91d in ardb::RedisRequestHandler::MessageReceived (e=..., ctx=..., this=0x7f64ad254f00) at network.cpp:130
#22 ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent (e=..., ctx=..., this=0x7f64ad254f00) at ./common/channel/channel_upstream_handler.hpp:133
#23 ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame> (ctx=<optimized out>, e=..., this=<optimized out>) at ./common/channel/all_includes.hpp:89
#24 0x00000000004d2680 in ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> > (this=0x7f64aca6de00, this=0x7f64aca6de00, e=...) at ./common/channel/all_includes.hpp:164
#25 ardb::fire_message_received<ardb::codec::RedisCommandFrame> (destructor=0x0, message=0x7f64929f3200, ctx=...) at ./common/channel/channel_helper.hpp:91
#26 ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode (this=0x7f64acb1e8c0, context=..., channel=0x7f64acb6c280, cumulation=...) at ./common/channel/codec/stack_frame_decoder.hpp:102
#27 0x00000000004d31c8 in ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived (this=0x7f64acb1e8c0, ctx=..., e=...) at ./common/channel/codec/stack_frame_decoder.hpp:157
#28 0x00000000004deb03 in ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent (e=..., ctx=..., this=<optimized out>) at ./common/channel/channel_upstream_handler.hpp:133
#29 ardb::ChannelPipeline::SendUpstream<ardb::Buffer> (ctx=<optimized out>, e=..., this=0x7f64acb6c2b8) at ./common/channel/all_includes.hpp:89
#30 0x00000000004dfdc2 in ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> > (event=..., this=0x7f64acb6c2b8) at ./common/channel/all_includes.hpp:128
---Type <return> to continue, or q <return> to quit---Quit
(gdb) f 1 
#1  0x00000000004d3d29 in ardb::Data::Compare (this=this@entry=0x7f6491f08d60, right=..., alpha_cmp=alpha_cmp@entry=false) at types.cpp:334
334 types.cpp: No such file or directory.
(gdb) info args
this = 0x7f6491f08d60
right = @0x7f64929f2e18: {data = 140069921666104, len = 2448240640, encoding = 100 'd'}
alpha_cmp = false
(gdb) print *
A syntax error in expression, near `'.
(gdb) print *this
$1 = {data = 140069921666152, len = 8, encoding = 4 '\004'}```

[shailesh33]

This seems like a combination of libc and jemalloc issue. Can you advice on how to build ardb with rocksdb using jemalloc?

[yinqiwen]

Is the server busy when this happened? seems the map m_locking_keys is corrupted. in the crash stack, the right is invalid.

[shailesh33]

no. even a basic hset key field value crashed. I am suspecting something wonky with the build

[yinqiwen]

make clean & make to rebuild again.