search

yinqiwen / ardb

A redis protocol compatible nosql, it support multiple storage engines as backend like Google's LevelDB, Facebook's RocksDB, OpenLDAP's LMDB, PerconaFT, WiredTiger, ForestDB.
BSD 3-Clause "New" or "Revised" License
1.83k stars 278 forks source link

segmentation fault 11 on running my unit tests #274

Closed 72squared closed 8 years ago

72squared commented 8 years ago

Last few lines of output on the server looks like:

[88313] 07-17 19:44:00,753 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size: 1073741824
[88313] 07-17 19:44:00,753 INFO [RocksDB]                  Options.table_properties_collectors: 
[88313] 07-17 19:44:00,753 INFO [RocksDB]                  Options.inplace_update_support: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[88313] 07-17 19:44:00,753 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[88313] 07-17 19:44:00,753 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[88313] 07-17 19:44:00,753 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]                          Options.bloom_locality: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]                   Options.max_successive_merges: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]               Options.paranoid_file_checks: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[88313] 07-17 19:44:00,755 INFO [RocksDB]Created column family [0] (ID 43)
[88313] 07-17 19:44:00,755 INFO Create ColumnFamilyHandle with name:0 success.
Segmentation fault: 11

Anything else I should do to gather info? It is very easy to repro, but I haven't isolated it to a certain command yet.

yinqiwen commented 8 years ago

can u use gdb with the core dump file to get the crash stack trace?
to get the core dump file ,u need to ulimit -c unlimited before start ardb-server in the same terminal shell.

72squared commented 8 years ago

Another variation (not sure if it is related or not) looks like this:

[88390] 07-17 19:48:51,157 INFO RocksDB drop column family:0.
[88390] 07-17 19:48:51,159 INFO [RocksDB]Dropped column family with id 67

ardb-server(88390,0x700007055000) malloc: *** error for object 0xffffffff: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

I'll look into gdb.

yinqiwen commented 8 years ago

seems u are using flushdb or flushall

72squared commented 8 years ago

Yes, my unit test suite depends on them to clear state at each test run. Are they unsupported commands?

yinqiwen commented 8 years ago

it's supported.

72squared commented 8 years ago

I was able to get the core dump file after a bit of research. I'm running this on my mac. Can try running it on ubuntu and see if it happens there too. The core dump file is 1GB and when I run it I get:

AL Decode second key prefix failed in comparator. 
Abort trap: 6 (core dumped)
jloehrer@jl-mac~/Code/ardb/src$ gdb ./ardb-server /cores/core.88797 
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin15.4.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ardb-server...done.
"/cores/core.88797": no core file handler recognizes format
(gdb)

So, not very useful. I'll switch to debugging in an ubuntu vm.

yinqiwen commented 8 years ago

just type bt to get the stack trace

yinqiwen commented 8 years ago

and in my mac, i use lldb ./ardb-server -c <core file> to debug

72squared commented 8 years ago

Here ya go:

jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.97974 
(lldb) target create "./src/ardb-server" --core "/cores/core.97974"
warning: (x86_64) /cores/core.97974 load command 148 LC_SEGMENT_64 has a fileoff + filesize (0x37ff8000) that extends beyond the end of the file (0x37ff7000), the segment will be truncated to match
warning: (x86_64) /cores/core.97974 load command 149 LC_SEGMENT_64 has a fileoff (0x37ff8000) that extends beyond the end of the file (0x37ff7000), ignoring this section
Core file '/cores/core.97974' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000102d804aa ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007fe983f00078, tvp=0x00007fff5cecc248) + 87 at ae_select.cc:48
    frame #2: 0x0000000102d80453 ardb-server`aeProcessEvents(eventLoop=0x00007fe983f00078, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x0000000102d8095b ardb-server`aeMain(eventLoop=0x00007fe983f00078) + 43 at ae.c:485
    frame #4: 0x0000000102d3d721 ardb-server`ardb::Server::Start(this=0x00007fff5cecc460) + 2417 at network.cpp:300
    frame #5: 0x0000000102d34858 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
    frame #7: 0x00007fff9c9e55ad libdyld.dylib`start + 1

Hope it helps. I'll try to narrow it down to a repeatable set of commands to cause the crash.

yinqiwen commented 8 years ago

can u use thread list to list all threads?

72squared commented 8 years ago 
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
yinqiwen commented 8 years ago

it's crashed in thread #25 where there is __pthread_kill, u can use thread select 25 to switch current thread to 25, and then bt to get the thread 25's stack trace.

72squared commented 8 years ago 
(lldb) thread select 25 
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff87cc8f06 <+10>: jae    0x7fff87cc8f10            ; <+20>
    0x7fff87cc8f08 <+12>: movq   %rax, %rdi
    0x7fff87cc8f0b <+15>: jmp    0x7fff87cc37cd            ; cerror_nocancel
    0x7fff87cc8f10 <+20>: retq   
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff95bdf4ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff959466e7 libsystem_c.dylib`abort + 129
    frame #3: 0x0000000102e19dd7 ardb-server`ardb::RocksDBPrefixExtractor::Transform(this=<unavailable>, src=<unavailable>) const + 247 at rocksdb_engine.cpp:385
    frame #4: 0x0000000102eb6682 ardb-server`rocksdb::MemTable::Get(this=0x00007fe985010c00, key=0x00007000048456e0, value="", s=0x00007fe983e05a60, merge_context=0x0000700004845698, seq=0x00007000048456d0) + 210 at memtable.cc:521
    frame #5: 0x0000000102e6d727 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) [inlined] rocksdb::MemTable::Get(this=<unavailable>, key=<unavailable>, value=<unavailable>, s=<unavailable>, merge_context=<unavailable>) + 1911 at memtable.h:175
    frame #6: 0x0000000102e6d704 ardb-server`rocksdb::DBImpl::MultiGet(this=0x00007fe98402ac00, read_options=0x00007000048458a8, column_family=0x0000700004845920, keys=0x0000700004845900, values=0x00007fe983e03c18 size=23) + 1876 at db_impl.cc:3199
    frame #7: 0x0000000102e14652 ardb-server`ardb::RocksDBEngine::MultiGet(this=<unavailable>, ctx=<unavailable>, keys=<unavailable>, values=size=23, errs=size=23) + 1010 at rocksdb_engine.cpp:1018
    frame #8: 0x0000000102dbae1f ardb-server`ardb::Ardb::HMGet(this=0x00007fff5cecc500, ctx=0x00007fe983c547f0, cmd=<unavailable>) + 655 at t_hash.cpp:324
    frame #9: 0x0000000102de7f92 ardb-server`ardb::Ardb::DoCall(this=0x00007fff5cecc500, ctx=0x00007fe983c547f0, setting=0x00007fe98403e7d8, args=0x0000700004845c68) + 258 at db.cpp:1395
    frame #10: 0x0000000102de870b ardb-server`ardb::Ardb::Call(this=0x00007fff5cecc500, ctx=0x00007fe983c547f0, args=0x0000700004845c68) + 1483 at db.cpp:1574
    frame #11: 0x0000000102d3e487 ardb-server`ardb::RedisRequestHandler::MessageReceived(this=0x00007fe983c54790, ctx=<unavailable>, e=<unavailable>) + 231 at network.cpp:86
    frame #12: 0x0000000102d5537a ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent(e=<unavailable>) + 506 at channel_upstream_handler.hpp:133
    frame #13: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::codec::RedisCommandFrame>&) at all_includes.hpp:89
    frame #14: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> >(this=<unavailable>) at all_includes.hpp:164
    frame #15: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<ardb::codec::RedisCommandFrame>(ctx=0x00007fe983c54690) at channel_helper.hpp:91
    frame #16: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(this=0x00007fe983c54650, context=0x00007fe983c54690, channel=0x00007fe983c54530, cumulation=0x00007fe983c54598) + 487 at stack_frame_decoder.hpp:102
    frame #17: 0x0000000102d532e7 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived(this=0x00007fe983c54650, ctx=<unavailable>, e=0x0000700004845db0) + 327 at stack_frame_decoder.hpp:157
    frame #18: 0x0000000102d43605 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #19: 0x0000000102d435f2 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #20: 0x0000000102d435b0 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #21: 0x0000000102d435a0 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x0000000103046f40) + 43 at channel_helper.hpp:83
    frame #22: 0x0000000102d43575 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #23: 0x0000000102d41a5e ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007fe983c54530, mask=1) + 30 at channel.cpp:53
    frame #24: 0x0000000102d80596 ardb-server`aeProcessEvents(eventLoop=0x00007fe983c53c78, flags=<unavailable>) + 630 at ae.c:429
    frame #25: 0x0000000102d8095b ardb-server`aeMain(eventLoop=0x00007fe983c53c78) + 43 at ae.c:485
    frame #26: 0x0000000102d6d295 ardb-server`ardb::Thread::ThreadFunc(data=0x00007fe983e01fc0) + 21 at thread.cpp:38
    frame #27: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #28: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #29: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
yinqiwen commented 8 years ago

it's an assert crash at https://github.com/yinqiwen/ardb/blob/0.9/src/db/rocksdb/rocksdb_engine.cpp#L385, seems the data is corrupted.

72squared commented 8 years ago

hmmm, so this is strange. I start fresh at the beginning of each test, and each test only lasts a second or so, and I run hundreds of them. The data types primarily used are sorted sets and hashes with very simple commands. All single key commands. But I'll see if I can narrow it down to a single command. So far when I restart the ardb-server and run a single test, it passes. but if I run multiple ones, it fails. Maybe something about doing flushall then right away writing again?

72squared commented 8 years ago

Would it help to have the database files? There's nothing confidential in it since it is just test data, so I can pass it along to you.

yinqiwen commented 8 years ago

The crash is related the data in memtable, so the data file maybe not helpful.
Is the unit test complated? maybe i can use to reproduce this issue in my mac.

yinqiwen commented 8 years ago

And in the stack trace, the crash is happened with hmget.

72squared commented 8 years ago

Let me see if I can come up with the simplest test to recreate the issue. hmget is probably one of the most common commands I run so that may not signify anything in particular.

72squared commented 8 years ago

I was able to repro the crash by running only three commands repeatedly from redis-cli over and over:

[MP]~/Code/happybits-devops/sites/marcopolo> redis-cli -p 16379
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> hgetall test1
1) "modified_by"
2) "test"
3) "resolved"
4) "False"
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> hgetall test1
(empty list or set)
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> hgetall test1
(empty list or set)
127.0.0.1:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
127.0.0.1:16379> hgetall test1
1) "modified_by"
2) "test"
3) "resolved"
4) "False"
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> hgetall test1
(empty list or set)
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> hgetall test1
(empty list or set)
127.0.0.1:16379> flushall
OK
127.0.0.1:16379> hgetall test1
Could not connect to Redis at 127.0.0.1:16379: Connection refused
not connected>

Here's the last few lines of ardb-server output:

[98921] 07-17 21:50:25,639 INFO [RocksDB]--------------- Options for column family [0]:

[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.comparator: rocksdb.InternalKeyComparator:ardb.comparator
[98921] 07-17 21:50:25,639 INFO [RocksDB]          Options.merge_operator: ardb.merger
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compaction_filter: None
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compaction_filter_factory: ardb.compact_filter_factory
[98921] 07-17 21:50:25,639 INFO [RocksDB]        Options.memtable_factory: SkipListFactory
[98921] 07-17 21:50:25,639 INFO [RocksDB]           Options.table_factory: BlockBasedTable
[98921] 07-17 21:50:25,639 INFO [RocksDB]           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fd3f2c4d320)
  cache_index_and_filter_blocks: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fd3f2c4e698
  block_cache_size: 536870912
  block_cache_compressed: 0x0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  skip_table_builder_flush: 0
  format_version: 0

[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.write_buffer_size: 134217728
[98921] 07-17 21:50:25,639 INFO [RocksDB] Options.max_write_buffer_number: 6
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[0]: NoCompression
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[1]: NoCompression
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[2]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[3]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[4]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[5]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[6]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[98921] 07-17 21:50:25,639 INFO [RocksDB]            Options.num_levels: 7
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[98921] 07-17 21:50:25,639 INFO [RocksDB]                 Options.compression_opts.level: -1
[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.compression_opts.strategy: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[98921] 07-17 21:50:25,639 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[98921] 07-17 21:50:25,639 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[98921] 07-17 21:50:25,639 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[98921] 07-17 21:50:25,639 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.source_compaction_factor: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[98921] 07-17 21:50:25,639 INFO [RocksDB]                       Options.arena_block_size: 16777216
[98921] 07-17 21:50:25,639 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[98921] 07-17 21:50:25,639 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.disable_auto_compactions: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                          Options.filter_deletes: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]                        Options.compaction_style: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                          Options.compaction_pri: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.max_merge_width: 4294967295
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size: 1073741824
[98921] 07-17 21:50:25,639 INFO [RocksDB]                  Options.table_properties_collectors: 
[98921] 07-17 21:50:25,639 INFO [RocksDB]                  Options.inplace_update_support: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[98921] 07-17 21:50:25,639 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[98921] 07-17 21:50:25,639 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                          Options.bloom_locality: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]                   Options.max_successive_merges: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.paranoid_file_checks: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[98921] 07-17 21:50:25,641 INFO [RocksDB]Created column family [0] (ID 161)
[98921] 07-17 21:50:25,641 INFO Create ColumnFamilyHandle with name:0 success.
[98921] 07-17 21:50:29,268 INFO RocksDB drop column family:0.
[98921] 07-17 21:50:29,272 INFO [RocksDB]Dropped column family with id 161

Segmentation fault: 11 (core dumped)
yinqiwen commented 8 years ago

It's fixed by latest commit.

72squared commented 8 years ago

With latest commit, I now get:

[99295] 07-18 06:21:51,732 INFO [RocksDB]--------------- Options for column family [0]:

[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.comparator: rocksdb.InternalKeyComparator:ardb.comparator
[99295] 07-18 06:21:51,732 INFO [RocksDB]          Options.merge_operator: ardb.merger
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compaction_filter: None
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compaction_filter_factory: ardb.compact_filter_factory
[99295] 07-18 06:21:51,732 INFO [RocksDB]        Options.memtable_factory: SkipListFactory
[99295] 07-18 06:21:51,732 INFO [RocksDB]           Options.table_factory: BlockBasedTable
[99295] 07-18 06:21:51,732 INFO [RocksDB]           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fea51c56d20)
  cache_index_and_filter_blocks: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fea51c58098
  block_cache_size: 536870912
  block_cache_compressed: 0x0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  skip_table_builder_flush: 0
  format_version: 0

[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.write_buffer_size: 134217728
[99295] 07-18 06:21:51,732 INFO [RocksDB] Options.max_write_buffer_number: 6
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[0]: NoCompression
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[1]: NoCompression
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[2]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[3]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[4]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[5]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[6]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[99295] 07-18 06:21:51,732 INFO [RocksDB]            Options.num_levels: 7
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[99295] 07-18 06:21:51,732 INFO [RocksDB]                 Options.compression_opts.level: -1
[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.compression_opts.strategy: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[99295] 07-18 06:21:51,732 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[99295] 07-18 06:21:51,732 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[99295] 07-18 06:21:51,732 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[99295] 07-18 06:21:51,732 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.source_compaction_factor: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[99295] 07-18 06:21:51,732 INFO [RocksDB]                       Options.arena_block_size: 16777216
[99295] 07-18 06:21:51,732 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[99295] 07-18 06:21:51,732 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.disable_auto_compactions: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                          Options.filter_deletes: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]                        Options.compaction_style: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                          Options.compaction_pri: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.max_merge_width: 4294967295
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size: 1073741824
[99295] 07-18 06:21:51,732 INFO [RocksDB]                  Options.table_properties_collectors: 
[99295] 07-18 06:21:51,732 INFO [RocksDB]                  Options.inplace_update_support: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[99295] 07-18 06:21:51,732 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[99295] 07-18 06:21:51,732 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                          Options.bloom_locality: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]                   Options.max_successive_merges: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.paranoid_file_checks: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[99295] 07-18 06:21:51,734 INFO [RocksDB]Created column family [0] (ID 175)
[99295] 07-18 06:21:51,734 INFO Create ColumnFamilyHandle with name:0 success.
[99295] 07-18 06:21:51,753 FATAL Not a valid key slice in PrefixExtractor with len:0
Abort trap: 6 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.99295 
(lldb) target create "./src/ardb-server" --core "/cores/core.99295"
warning: (x86_64) /cores/core.99295 load command 151 LC_SEGMENT_64 has a fileoff + filesize (0x390f7000) that extends beyond the end of the file (0x390f6000), the segment will be truncated to match
warning: (x86_64) /cores/core.99295 load command 152 LC_SEGMENT_64 has a fileoff (0x390f7000) that extends beyond the end of the file (0x390f6000), ignoring this section
Core file '/cores/core.99295' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x000000010a2f9c6a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007fea53000118, tvp=0x00007fff55952248) + 87 at ae_select.cc:48
    frame #2: 0x000000010a2f9c13 ardb-server`aeProcessEvents(eventLoop=0x00007fea53000118, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x000000010a2fa11b ardb-server`aeMain(eventLoop=0x00007fea53000118) + 43 at ae.c:485
    frame #4: 0x000000010a2b6ee1 ardb-server`ardb::Server::Start(this=0x00007fff55952460) + 2417 at network.cpp:300
    frame #5: 0x000000010a2ae018 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
(lldb) thread select 26
* thread #26: tid = 0x0019, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff87cc8f06 <+10>: jae    0x7fff87cc8f10            ; <+20>
    0x7fff87cc8f08 <+12>: movq   %rax, %rdi
    0x7fff87cc8f0b <+15>: jmp    0x7fff87cc37cd            ; cerror_nocancel
    0x7fff87cc8f10 <+20>: retq   
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #26: tid = 0x0019, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff95bdf4ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff959466e7 libsystem_c.dylib`abort + 129
    frame #3: 0x000000010a392e92 ardb-server`ardb::RocksDBPrefixExtractor::Transform(this=<unavailable>, src=<unavailable>) const + 162 at rocksdb_engine.cpp:418
    frame #4: 0x000000010a42f782 ardb-server`rocksdb::MemTable::Get(this=0x00007fea5400aa00, key=0x00007000050486e0, value="", s=0x00007fea51e03d90, merge_context=0x0000700005048698, seq=0x00007000050486d0) + 210 at memtable.cc:521
    frame #5: 0x000000010a3e6827 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) [inlined] rocksdb::MemTable::Get(this=<unavailable>, key=<unavailable>, value=<unavailable>, s=<unavailable>, merge_context=<unavailable>) + 1911 at memtable.h:175
    frame #6: 0x000000010a3e6804 ardb-server`rocksdb::DBImpl::MultiGet(this=0x00007fea5202ac00, read_options=0x00007000050488a8, column_family=0x0000700005048920, keys=0x0000700005048900, values=0x00007fea51c5bc88 size=23) + 1876 at db_impl.cc:3199
    frame #7: 0x000000010a38ddb2 ardb-server`ardb::RocksDBEngine::MultiGet(this=<unavailable>, ctx=<unavailable>, keys=<unavailable>, values=size=23, errs=size=23) + 1010 at rocksdb_engine.cpp:1051
    frame #8: 0x000000010a3345df ardb-server`ardb::Ardb::HMGet(this=0x00007fff55952500, ctx=0x00007fea51c59b30, cmd=<unavailable>) + 655 at t_hash.cpp:325
    frame #9: 0x000000010a361752 ardb-server`ardb::Ardb::DoCall(this=0x00007fff55952500, ctx=0x00007fea51c59b30, setting=0x00007fea5203e7d8, args=0x0000700005048c68) + 258 at db.cpp:1395
    frame #10: 0x000000010a361ecb ardb-server`ardb::Ardb::Call(this=0x00007fff55952500, ctx=0x00007fea51c59b30, args=0x0000700005048c68) + 1483 at db.cpp:1574
    frame #11: 0x000000010a2b7c47 ardb-server`ardb::RedisRequestHandler::MessageReceived(this=0x00007fea51c59ad0, ctx=<unavailable>, e=<unavailable>) + 231 at network.cpp:86
    frame #12: 0x000000010a2ceb3a ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent(e=<unavailable>) + 506 at channel_upstream_handler.hpp:133
    frame #13: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::codec::RedisCommandFrame>&) at all_includes.hpp:89
    frame #14: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> >(this=<unavailable>) at all_includes.hpp:164
    frame #15: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<ardb::codec::RedisCommandFrame>(ctx=0x00007fea51c5adf0) at channel_helper.hpp:91
    frame #16: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(this=0x00007fea51c5ba60, context=0x00007fea51c5adf0, channel=0x00007fea51c5b940, cumulation=0x00007fea51c5b9a8) + 487 at stack_frame_decoder.hpp:102
    frame #17: 0x000000010a2ccaa7 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived(this=0x00007fea51c5ba60, ctx=<unavailable>, e=0x0000700005048db0) + 327 at stack_frame_decoder.hpp:157
    frame #18: 0x000000010a2bcdc5 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #19: 0x000000010a2bcdb2 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #20: 0x000000010a2bcd70 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #21: 0x000000010a2bcd60 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x000000010a5bff40) + 43 at channel_helper.hpp:83
    frame #22: 0x000000010a2bcd35 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #23: 0x000000010a2bb21e ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007fea51c5b940, mask=1) + 30 at channel.cpp:53
    frame #24: 0x000000010a2f9d56 ardb-server`aeProcessEvents(eventLoop=0x00007fea51e02098, flags=<unavailable>) + 630 at ae.c:429
    frame #25: 0x000000010a2fa11b ardb-server`aeMain(eventLoop=0x00007fea51e02098) + 43 at ae.c:485
    frame #26: 0x000000010a2e6a55 ardb-server`ardb::Thread::ThreadFunc(data=0x00007fea51c53990) + 21 at thread.cpp:38
    frame #27: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #28: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #29: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
(lldb)
yinqiwen commented 8 years ago

seems there is a bug for empty key in rocksdb. fixed in last commit.
is there any hsetnx/setnx in your unit test? setnx/hetnx may generate empty keys in rocksdb.

72squared commented 8 years ago

yes, it is used frequently.

72squared commented 8 years ago

okay that latest commit allowed one set of previously failing unit tests to pass. Here's the next core dump I get:

[201] 07-18 06:44:35,087 INFO [RocksDB]--------------- Options for column family [0]:

[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.comparator: rocksdb.InternalKeyComparator:ardb.comparator
[201] 07-18 06:44:35,087 INFO [RocksDB]          Options.merge_operator: ardb.merger
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compaction_filter: None
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compaction_filter_factory: ardb.compact_filter_factory
[201] 07-18 06:44:35,087 INFO [RocksDB]        Options.memtable_factory: SkipListFactory
[201] 07-18 06:44:35,087 INFO [RocksDB]           Options.table_factory: BlockBasedTable
[201] 07-18 06:44:35,087 INFO [RocksDB]           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f91d2c56d20)
  cache_index_and_filter_blocks: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f91d2c58098
  block_cache_size: 536870912
  block_cache_compressed: 0x0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  skip_table_builder_flush: 0
  format_version: 0

[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.write_buffer_size: 134217728
[201] 07-18 06:44:35,087 INFO [RocksDB] Options.max_write_buffer_number: 6
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[0]: NoCompression
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[1]: NoCompression
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[2]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[3]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[4]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[5]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[6]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[201] 07-18 06:44:35,087 INFO [RocksDB]            Options.num_levels: 7
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[201] 07-18 06:44:35,087 INFO [RocksDB]                 Options.compression_opts.level: -1
[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.compression_opts.strategy: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[201] 07-18 06:44:35,087 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[201] 07-18 06:44:35,087 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[201] 07-18 06:44:35,087 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[201] 07-18 06:44:35,087 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.source_compaction_factor: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[201] 07-18 06:44:35,087 INFO [RocksDB]                       Options.arena_block_size: 16777216
[201] 07-18 06:44:35,087 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[201] 07-18 06:44:35,087 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.disable_auto_compactions: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                          Options.filter_deletes: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]                        Options.compaction_style: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                          Options.compaction_pri: 0
[201] 07-18 06:44:35,087 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.max_merge_width: 4294967295
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size: 1073741824
[201] 07-18 06:44:35,087 INFO [RocksDB]                  Options.table_properties_collectors: 
[201] 07-18 06:44:35,087 INFO [RocksDB]                  Options.inplace_update_support: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[201] 07-18 06:44:35,087 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[201] 07-18 06:44:35,087 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                          Options.bloom_locality: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]                   Options.max_successive_merges: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.paranoid_file_checks: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[201] 07-18 06:44:35,089 INFO [RocksDB]Created column family [0] (ID 245)
[201] 07-18 06:44:35,089 INFO Create ColumnFamilyHandle with name:0 success.
Segmentation fault: 11 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.201 
(lldb) target create "./src/ardb-server" --core "/cores/core.201"
warning: (x86_64) /cores/core.201 load command 152 LC_SEGMENT_64 has a fileoff + filesize (0x398f7000) that extends beyond the end of the file (0x398f6000), the segment will be truncated to match
warning: (x86_64) /cores/core.201 load command 153 LC_SEGMENT_64 has a fileoff (0x398f7000) that extends beyond the end of the file (0x398f6000), ignoring this section
Core file '/cores/core.201' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x000000010132fc4a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007f91d2c07ea8, tvp=0x00007fff5e91c248) + 87 at ae_select.cc:48
    frame #2: 0x000000010132fbf3 ardb-server`aeProcessEvents(eventLoop=0x00007f91d2c07ea8, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x00000001013300fb ardb-server`aeMain(eventLoop=0x00007f91d2c07ea8) + 43 at ae.c:485
    frame #4: 0x00000001012ecec1 ardb-server`ardb::Server::Start(this=0x00007fff5e91c460) + 2417 at network.cpp:300
    frame #5: 0x00000001012e3ff8 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff9593b76e libsystem_c.dylib`_vsnprintf + 173, stop reason = signal SIGSTOP
(lldb) thread select 25
* thread #25: tid = 0x0018, 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122, stop reason = signal SIGSTOP
    frame #0: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122
   119              }
   120              inline bool Readable() const
   121              {
-> 122                  return m_write_idx > m_read_idx;
   123              }
   124              inline bool Writeable() const
   125              {
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #25: tid = 0x0018, 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122, stop reason = signal SIGSTOP
  * frame #0: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122
    frame #1: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::ReadableBytes() const at buffer.hpp:130
    frame #2: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ch=0x00007f91d50002e0, data=0x7000000000000000) + 52 at server.cpp:1021
    frame #3: 0x00000001012f77f3 ardb-server`ardb::ChannelService::OnSoftSignal(this=0x00007f91d2e02c10, soft_signo=<unavailable>, appendinfo=<unavailable>) + 67 at channel_service.cpp:175
    frame #4: 0x000000010130ec33 ardb-server`ardb::SoftSignalChannel::FireSignalReceived(this=<unavailable>, signo=4, append_info=1) + 243 at soft_signal_channel.cpp:72
    frame #5: 0x000000010130f764 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<unsigned long long>::HandleStreamEvent(e=<unavailable>) + 292 at channel_upstream_handler.hpp:133
    frame #6: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<unsigned long long>(ardb::ChannelHandlerContext*, ardb::MessageEvent<unsigned long long>&) at all_includes.hpp:89
    frame #7: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<unsigned long long> >(this=<unavailable>) at all_includes.hpp:164
    frame #8: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<unsigned long long>(ctx=0x00007f91d2e02b50) at channel_helper.hpp:91
    frame #9: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(this=0x00007f91d2e03058, context=0x00007f91d2e02b50, channel=0x00007f91d2e02f40, cumulation=0x00007f91d2e02fa8) + 273 at stack_frame_decoder.hpp:102
    frame #10: 0x000000010130f2b7 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::MessageReceived(this=0x00007f91d2e03058, ctx=<unavailable>, e=0x0000700004845db0) + 327 at stack_frame_decoder.hpp:157
    frame #11: 0x00000001012f2da5 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #12: 0x00000001012f2d92 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #13: 0x00000001012f2d50 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #14: 0x00000001012f2d40 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x00000001015f5f40) + 43 at channel_helper.hpp:83
    frame #15: 0x00000001012f2d15 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #16: 0x00000001012f11fe ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007f91d2e02f40, mask=1) + 30 at channel.cpp:53
    frame #17: 0x000000010132fd36 ardb-server`aeProcessEvents(eventLoop=0x00007f91d2e02b08, flags=<unavailable>) + 630 at ae.c:429
    frame #18: 0x00000001013300fb ardb-server`aeMain(eventLoop=0x00007f91d2e02b08) + 43 at ae.c:485
    frame #19: 0x000000010131ca35 ardb-server`ardb::Thread::ThreadFunc(data=0x00007f91d2e02bd0) + 21 at thread.cpp:38
    frame #20: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #21: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #22: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
(lldb)
yinqiwen commented 8 years ago

is it crashed with monitor command?

72squared commented 8 years ago

I was running monitor in another window, but that doesn't occur in my unit tests anywhere. I'll re-run the tests with no monitor.

72squared commented 8 years ago

here you go:

[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.write_buffer_size: 134217728
[324] 07-18 06:59:49,159 INFO [RocksDB] Options.max_write_buffer_number: 6
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[0]: NoCompression
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[1]: NoCompression
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[2]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[3]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[4]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[5]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[6]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[324] 07-18 06:59:49,159 INFO [RocksDB]            Options.num_levels: 7
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[324] 07-18 06:59:49,159 INFO [RocksDB]                 Options.compression_opts.level: -1
[324] 07-18 06:59:49,159 INFO [RocksDB]              Options.compression_opts.strategy: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[324] 07-18 06:59:49,159 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[324] 07-18 06:59:49,159 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[324] 07-18 06:59:49,159 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[324] 07-18 06:59:49,159 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.source_compaction_factor: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[324] 07-18 06:59:49,159 INFO [RocksDB]                       Options.arena_block_size: 16777216
[324] 07-18 06:59:49,159 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[324] 07-18 06:59:49,159 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.disable_auto_compactions: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                          Options.filter_deletes: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]                        Options.compaction_style: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                          Options.compaction_pri: 0
[324] 07-18 06:59:49,159 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.max_merge_width: 4294967295
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size: 1073741824
[324] 07-18 06:59:49,159 INFO [RocksDB]                  Options.table_properties_collectors: 
[324] 07-18 06:59:49,159 INFO [RocksDB]                  Options.inplace_update_support: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[324] 07-18 06:59:49,159 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[324] 07-18 06:59:49,159 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[324] 07-18 06:59:49,159 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                          Options.bloom_locality: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]                   Options.max_successive_merges: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.paranoid_file_checks: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[324] 07-18 06:59:49,162 INFO [RocksDB]Created column family [0] (ID 246)
[324] 07-18 06:59:49,162 INFO Create ColumnFamilyHandle with name:0 success.
Segmentation fault: 11 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb src/ardb-server -c /cores/core.
core.201  core.324  
jloehrer@jl-mac~/Code/ardb$ lldb src/ardb-server -c /cores/core.324 
(lldb) target create "src/ardb-server" --core "/cores/core.324"
warning: (x86_64) /cores/core.324 load command 148 LC_SEGMENT_64 has a fileoff + filesize (0x37ff7000) that extends beyond the end of the file (0x37ff6000), the segment will be truncated to match
warning: (x86_64) /cores/core.324 load command 149 LC_SEGMENT_64 has a fileoff (0x37ff7000) that extends beyond the end of the file (0x37ff6000), ignoring this section
Core file '/cores/core.324' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000103996c4a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007fc601d016f8, tvp=0x00007fff5c2b5248) + 87 at ae_select.cc:48
    frame #2: 0x0000000103996bf3 ardb-server`aeProcessEvents(eventLoop=0x00007fc601d016f8, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x00000001039970fb ardb-server`aeMain(eventLoop=0x00007fc601d016f8) + 43 at ae.c:485
    frame #4: 0x0000000103953ec1 ardb-server`ardb::Server::Start(this=0x00007fff5c2b5460) + 2417 at network.cpp:300
    frame #5: 0x000000010394aff8 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x0000000103a837f0 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) + 20 at __hash_table:2020, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
(lldb) thread select 25
* thread #25: tid = 0x0018, 0x0000000103a837f0 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) + 20 at __hash_table:2020, stop reason = signal SIGSTOP
    frame #0: 0x0000000103a837f0 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) + 20 at __hash_table:2020
   2017         __node_pointer __nd = __bucket_list_[__chash];
   2018         if (__nd != nullptr)
   2019         {
-> 2020             for (__nd = __nd->__next_; __nd != nullptr &&
   2021                                        __constrain_hash(__nd->__hash_, __bc) == __chash;
   2022                                                            __nd = __nd->__next_)
   2023             {
(lldb)
yinqiwen commented 8 years ago

it's crashed in rocksdb's code now.
can u make clean and then rebuild to test again?

72squared commented 8 years ago

did make clean and then storage_engine=rocksdb make. I also removed the data dir to start fresh, then started it up and ran my tests:

[1161] 07-18 07:52:26,190 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size: 1073741824
[1161] 07-18 07:52:26,190 INFO [RocksDB]                  Options.table_properties_collectors: 
[1161] 07-18 07:52:26,190 INFO [RocksDB]                  Options.inplace_update_support: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[1161] 07-18 07:52:26,190 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[1161] 07-18 07:52:26,190 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[1161] 07-18 07:52:26,190 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[1161] 07-18 07:52:26,190 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]                          Options.bloom_locality: 1
[1161] 07-18 07:52:26,190 INFO [RocksDB]                   Options.max_successive_merges: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]               Options.paranoid_file_checks: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[1161] 07-18 07:52:26,192 INFO [RocksDB]Created column family [0] (ID 37)
[1161] 07-18 07:52:26,192 INFO Create ColumnFamilyHandle with name:0 success.
[1161] 07-18 07:52:26,211 FATAL Not a valid key slice in PrefixExtractor with len:39
Abort trap: 6 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.11
core.1115  core.1126  core.1161  
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.1161 
(lldb) target create "./src/ardb-server" --core "/cores/core.1161"
warning: (x86_64) /cores/core.1161 load command 152 LC_SEGMENT_64 has a fileoff + filesize (0x398f7000) that extends beyond the end of the file (0x398f6000), the segment will be truncated to match
warning: (x86_64) /cores/core.1161 load command 153 LC_SEGMENT_64 has a fileoff (0x398f7000) that extends beyond the end of the file (0x398f6000), ignoring this section
Core file '/cores/core.1161' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000104822c4a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007f8313000158, tvp=0x00007fff5b429248) + 87 at ae_select.cc:48
    frame #2: 0x0000000104822bf3 ardb-server`aeProcessEvents(eventLoop=0x00007f8313000158, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x00000001048230fb ardb-server`aeMain(eventLoop=0x00007f8313000158) + 43 at ae.c:485
    frame #4: 0x00000001047dfec1 ardb-server`ardb::Server::Start(this=0x00007fff5b429460) + 2417 at network.cpp:300
    frame #5: 0x00000001047d6ff8 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
(lldb) thread select 25
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff87cc8f06 <+10>: jae    0x7fff87cc8f10            ; <+20>
    0x7fff87cc8f08 <+12>: movq   %rax, %rdi
    0x7fff87cc8f0b <+15>: jmp    0x7fff87cc37cd            ; cerror_nocancel
    0x7fff87cc8f10 <+20>: retq   
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff95bdf4ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff959466e7 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001048bbe7d ardb-server`ardb::RocksDBPrefixExtractor::Transform(this=<unavailable>, src=<unavailable>) const + 173 at rocksdb_engine.cpp:422
    frame #4: 0x0000000104958782 ardb-server`rocksdb::MemTable::Get(this=0x00007f8311034000, key=0x00007000048456e0, value="", s=0x00007f83130033a0, merge_context=0x0000700004845698, seq=0x00007000048456d0) + 210 at memtable.cc:521
    frame #5: 0x000000010490f827 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) [inlined] rocksdb::MemTable::Get(this=<unavailable>, key=<unavailable>, value=<unavailable>, s=<unavailable>, merge_context=<unavailable>) + 1911 at memtable.h:175
    frame #6: 0x000000010490f804 ardb-server`rocksdb::DBImpl::MultiGet(this=0x00007f831102b400, read_options=0x00007000048458a8, column_family=0x0000700004845920, keys=0x0000700004845900, values=0x00007f8310e02048 size=11) + 1876 at db_impl.cc:3199
    frame #7: 0x00000001048b6d92 ardb-server`ardb::RocksDBEngine::MultiGet(this=<unavailable>, ctx=<unavailable>, keys=<unavailable>, values=size=11, errs=size=11) + 1010 at rocksdb_engine.cpp:1055
    frame #8: 0x000000010485d5bf ardb-server`ardb::Ardb::HMGet(this=0x00007fff5b429500, ctx=0x00007f8310d02940, cmd=<unavailable>) + 655 at t_hash.cpp:325
    frame #9: 0x000000010488a732 ardb-server`ardb::Ardb::DoCall(this=0x00007fff5b429500, ctx=0x00007f8310d02940, setting=0x00007f831103e7d8, args=0x0000700004845c68) + 258 at db.cpp:1395
    frame #10: 0x000000010488aeab ardb-server`ardb::Ardb::Call(this=0x00007fff5b429500, ctx=0x00007f8310d02940, args=0x0000700004845c68) + 1483 at db.cpp:1574
    frame #11: 0x00000001047e0c27 ardb-server`ardb::RedisRequestHandler::MessageReceived(this=0x00007f8310d028e0, ctx=<unavailable>, e=<unavailable>) + 231 at network.cpp:86
    frame #12: 0x00000001047f7b1a ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent(e=<unavailable>) + 506 at channel_upstream_handler.hpp:133
    frame #13: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::codec::RedisCommandFrame>&) at all_includes.hpp:89
    frame #14: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> >(this=<unavailable>) at all_includes.hpp:164
    frame #15: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<ardb::codec::RedisCommandFrame>(ctx=0x00007f8310d027c0) at channel_helper.hpp:91
    frame #16: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(this=0x00007f8310d02780, context=0x00007f8310d027c0, channel=0x00007f8310d02660, cumulation=0x00007f8310d026c8) + 487 at stack_frame_decoder.hpp:102
    frame #17: 0x00000001047f5a87 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived(this=0x00007f8310d02780, ctx=<unavailable>, e=0x0000700004845db0) + 327 at stack_frame_decoder.hpp:157
    frame #18: 0x00000001047e5da5 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #19: 0x00000001047e5d92 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #20: 0x00000001047e5d50 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #21: 0x00000001047e5d40 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x0000000104ae8f40) + 43 at channel_helper.hpp:83
    frame #22: 0x00000001047e5d15 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #23: 0x00000001047e41fe ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007f8310d02660, mask=1) + 30 at channel.cpp:53
    frame #24: 0x0000000104822d36 ardb-server`aeProcessEvents(eventLoop=0x00007f8310c54648, flags=<unavailable>) + 630 at ae.c:429
    frame #25: 0x00000001048230fb ardb-server`aeMain(eventLoop=0x00007f8310c54648) + 43 at ae.c:485
    frame #26: 0x000000010480fa35 ardb-server`ardb::Thread::ThreadFunc(data=0x00007f8310c56f40) + 21 at thread.cpp:38
    frame #27: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #28: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #29: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
(lldb)
yinqiwen commented 8 years ago

what commands used before hmget test?
there is FATAL Not a valid key slice in PrefixExtractor with len:39 in log which indicate that the stored key is not valid.

72squared commented 8 years ago

I re-ran the test with monitor connected. Here's the output, but I'm not sure how much it will help. I'll try to find a more minimal set of commands to repro the issue.

1468855106.170175 [0 127.0.0.1:62135] "flushall" 
1468855106.171117 [0 127.0.0.1:62135] "hget" "U:phone:1:u:" "+18195554321"
1468855106.171720 [0 127.0.0.1:62135] "hmset" "D{bEdQGyj1TUaz7uFT3mNeJA}" "user_id" "3aOChHM5RMaGGz2cyjzUBQ" "locale" "en_US" "created_at" "1468855106" "app_type" "mpk" "platform_version" "16" "platform_type" "android" "secret" "sha256$647e1f2b00ef$526c490dbc91aa206bdddc4938bede1e28f9231f0c6b32ee37dd71a736af1eba" "last_login" "1468855106" "push_token" "110ef9c637ba6c14a884c917c703819f" "timezone" "UTC" "flavor" "beta" "app_version" "1.1.0" "device_id" "bEdQGyj1TUaz7uFT3mNeJA"
1468855106.174429 [0 127.0.0.1:62135] "hmset" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "James" "last_name" "Test-ofsky" "user_id" "3aOChHM5RMaGGz2cyjzUBQ" "signup_at" "1468768706" "phone" "+18195554321" "email" "test+erlamuyj@happybits.co"
1468855106.174743 [0 127.0.0.1:62135] "hsetnx" "U:phone:1:u:" "+18195554321" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.174928 [0 127.0.0.1:62135] "zadd" "T-U{user_set_test_users}" "1468855106.174859" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.175141 [0 127.0.0.1:62135] "zadd" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106" "bEdQGyj1TUaz7uFT3mNeJA"
1468855106.175508 [0 127.0.0.1:62135] "zrevrange" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "0" "0"
1468855106.175783 [0 127.0.0.1:62135] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.176228 [0 127.0.0.1:62135] "hgetall" "UAT{3aOChHM5RMaGGz2cyjzUBQ}"
1468855106.176360 [0 127.0.0.1:62135] "hgetall" "UEIBT{3aOChHM5RMaGGz2cyjzUBQ-invite-hybrid}"
1468855106.176600 [0 127.0.0.1:62135] "del" "SSMS{+18195554321}"
1468855106.176942 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_opens" "1"
1468855106.177043 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "activity_alerts" "on"
1468855106.177086 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_records" "0"
1468855106.177126 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_groups" "0"
1468855106.177180 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "current_orbit" "signup"
1468855106.177217 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "signup_tracked" "1"
1468855106.177273 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_friends" "0"
1468855106.177313 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "invited" "0"
1468855106.177355 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_recordDays" "0"
1468855106.177420 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_newplays" "0"
1468855106.177462 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "signup_time" "1468768706"
1468855106.177525 [0 127.0.0.1:62135] "hset" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "signup_tz" "UTC"
1468855106.177736 [0 127.0.0.1:62135] "zrange" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "0" "-1"
1468855106.178274 [0 127.0.0.1:62135] "zrange" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "0" "-1"
1468855106.178884 [0 127.0.0.1:62135] "hget" "U:phone:17:u:" "+17855554005"
1468855106.179353 [0 127.0.0.1:62135] "hsetnx" "ExperimentNameIndex:57:u" "IaLs5" "4nnrrGcLRIKdNcVgtsHosA"
1468855106.180033 [0 127.0.0.1:62135] "hmset" "ABEXP{4nnrrGcLRIKdNcVgtsHosA}" "name" "IaLs5" "recruiting" "1" "created_at" "1468855106" "modified_at" "1468855106" "recruiting_start" "1468855106" "exp_type" "invite-hybrid" "recruiting_location" "signup" "defaults" "{}" "active" "1" "exp_id" "4nnrrGcLRIKdNcVgtsHosA"
1468855106.180180 [0 127.0.0.1:62135] "zadd" "ABEXPRBT{invite-hybrid}" "1468855106.0" "4nnrrGcLRIKdNcVgtsHosA"
1468855106.180277 [0 127.0.0.1:62135] "zadd" "ABEXPL{all}" "1468855106" "4nnrrGcLRIKdNcVgtsHosA"
1468855106.181209 [0 127.0.0.1:62135] "hmset" "ABEXPC{dqIVHNuoTdy8_ZSraeL7Xw}" "name" "variant" "cohort_id" "dqIVHNuoTdy8_ZSraeL7Xw" "created_at" "1468855106" "modified_at" "1468855106" "exp_id" "4nnrrGcLRIKdNcVgtsHosA" "overrides" "{\"sms_message\": \"{{name}} {{url}}\", \"client_sms_message\": \"individual {{url}}\", \"client_invite_all_sms_message\": \"invite_all_client {{url}}\", \"client_sms_video_message\": \"individual _video_ {{url}}\", \"client_reinvite_sms_video_message\": \"reinvite_client _video_ {{url}}\", \"sms\": \"yes\", \"client_reinvite_sms_message\": \"reinvite_client {{url}}\", \"sms_video_message\": \"_video_ from {{name}} {{url}}\", \"client_suggested_sms_video_message\": \"suggested_client _video_ {{url}}\", \"client_invite_all_sms_video_message\": \"invite_all_client _video_ {{url}}\", \"client_suggested_sms_message\": \"suggested_client {{url}}\"}"
1468855106.181445 [0 127.0.0.1:62135] "zadd" "ABEXPCL{4nnrrGcLRIKdNcVgtsHosA}" "1468855106.180933" "dqIVHNuoTdy8_ZSraeL7Xw"
1468855106.183785 [0 127.0.0.1:62135] "hmget" "V{2YhxfUejSxyEhTRkxdkd3Q}" "status" "container" "deleted" "created_at" "video_id" "video_type" "upload_pending" "upload_server" "upload_service" "path_secret" "user_id" "local_upload_complete"
1468855106.185013 [0 127.0.0.1:62135] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.185250 [0 127.0.0.1:62135] "hmget" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.185740 [0 127.0.0.1:62136] "hmget" "M{TXKwyOL1SROI_ENeFttiPw}" "by" "deleted" "text" "created_at" "video_id" "modified_at" "emotes" "event" "creator_id" "conversation_id" "message_id"
1468855106.187384 [0 127.0.0.1:62136] "hget" "U:phone:17:u:" "+17855554005"
1468855106.187758 [0 127.0.0.1:62136] "hsetnx" "U:phone:17:u:" "+17855554005" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.187836 [0 127.0.0.1:62136] "hget" "U:phone:17:u:" "+17855554005"
1468855106.188207 [0 127.0.0.1:62136] "hmset" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "phone" "+17855554005" "user_id" "4p8_3B94TZyqDJ0-VAUS8Q" "email" "foogtebgpnv@gmail.com"
1468855106.188573 [0 127.0.0.1:62136] "zrange" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "0" "-1"
1468855106.188590 [0 127.0.0.1:62136] "zrange" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "-1"
1468855106.188944 [0 127.0.0.1:62136] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.189194 [0 127.0.0.1:62136] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.189538 [0 127.0.0.1:62136] "hmset" "C{oYyu_NBS7CAMoebwh4nVSw}" "conversation_id" "oYyu_NBS7CAMoebwh4nVSw" "created_at" "1468855106" "modified_at" "1468855106"
1468855106.189587 [0 127.0.0.1:62136] "hdel" "C{oYyu_NBS7CAMoebwh4nVSw}" "title"
1468855106.189924 [0 127.0.0.1:62136] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "-1"
1468855106.189941 [0 127.0.0.1:62136] "zscore" "UBL{4p8_3B94TZyqDJ0-VAUS8Q}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.189951 [0 127.0.0.1:62136] "zscore" "UBL{3aOChHM5RMaGGz2cyjzUBQ}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.189961 [0 127.0.0.1:62136] "hget" "UCRM{4p8_3B94TZyqDJ0-VAUS8Q}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.189978 [0 127.0.0.1:62136] "hget" "UCRM{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.190266 [0 127.0.0.1:62136] "zrevrange" "UD{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "0"
1468855106.190537 [0 127.0.0.1:62136] "hgetall" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}"
1468855106.190703 [0 127.0.0.1:62136] "hgetall" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}"
1468855106.190960 [0 127.0.0.1:62136] "zrevrange" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "0" "0"
1468855106.191173 [0 127.0.0.1:62136] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.191602 [0 127.0.0.1:62136] "hgetall" "UAT{3aOChHM5RMaGGz2cyjzUBQ}"
1468855106.191886 [0 127.0.0.1:62136] "hgetall" "UEIBT{3aOChHM5RMaGGz2cyjzUBQ-invite-hybrid}"
1468855106.192292 [0 127.0.0.1:62136] "zadd" "CM{oYyu_NBS7CAMoebwh4nVSw}" "1468855106.190096" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.192371 [0 127.0.0.1:62136] "zrem" "UCR{4p8_3B94TZyqDJ0-VAUS8Q}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.192396 [0 127.0.0.1:62136] "hdel" "UCRM{4p8_3B94TZyqDJ0-VAUS8Q}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.192408 [0 127.0.0.1:62136] "zadd" "CM{oYyu_NBS7CAMoebwh4nVSw}" "1468855106.190096" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.192460 [0 127.0.0.1:62136] "zrem" "UCR{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.192494 [0 127.0.0.1:62136] "hdel" "UCRM{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.192505 [0 127.0.0.1:62136] "zadd" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "1468855106.192079" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.192553 [0 127.0.0.1:62136] "zadd" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106.192079" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.193508 [0 127.0.0.1:62136] "hmget" "V{2YhxfUejSxyEhTRkxdkd3Q}" "status" "container" "deleted" "created_at" "video_id" "video_type" "upload_pending" "upload_server" "upload_service" "path_secret" "user_id" "local_upload_complete"
1468855106.194158 [0 127.0.0.1:62136] "hmset" "V{2YhxfUejSxyEhTRkxdkd3Q}" "container" "20160718" "created_at" "1468855106" "video_id" "2YhxfUejSxyEhTRkxdkd3Q" "video_type" "mp4" "upload_pending" "1" "upload_service" "speedys3" "path_secret" "hkfeco1dqywui0sjzaq5ha" "user_id" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.195257 [0 127.0.0.1:62136] "hmget" "M{TXKwyOL1SROI_ENeFttiPw}" "by" "deleted" "text" "created_at" "video_id" "modified_at" "emotes" "event" "creator_id" "conversation_id" "message_id"
1468855106.195511 [0 127.0.0.1:62136] "hmget" "M{TXKwyOL1SROI_ENeFttiPw}" "by" "deleted" "text" "created_at" "video_id" "modified_at" "emotes" "event" "creator_id" "conversation_id" "message_id"
1468855106.195783 [0 127.0.0.1:62136] "zrank" "CM{oYyu_NBS7CAMoebwh4nVSw}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.196241 [0 127.0.0.1:62136] "persist" "V{2YhxfUejSxyEhTRkxdkd3Q}"
1468855106.196536 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "500"
1468855106.196948 [0 127.0.0.1:62135] "zadd" "VM{2YhxfUejSxyEhTRkxdkd3Q}" "1468855106.195147" "TXKwyOL1SROI_ENeFttiPw"
1468855106.196996 [0 127.0.0.1:62135] "zadd" "CMA{oYyu_NBS7CAMoebwh4nVSw}" "1468855106.196229" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.197035 [0 127.0.0.1:62135] "zremrangebyscore" "CMA{oYyu_NBS7CAMoebwh4nVSw}" "-inf" "1468768706.196254"
1468855106.197072 [0 127.0.0.1:62135] "expire" "CMA{oYyu_NBS7CAMoebwh4nVSw}" "86401"
1468855106.197108 [0 127.0.0.1:62135] "zrevrangebyscore" "CMA{oYyu_NBS7CAMoebwh4nVSw}" "+inf" "1468768706.196323" "LIMIT" "0" "2"
1468855106.197152 [0 127.0.0.1:62135] "hmset" "M{TXKwyOL1SROI_ENeFttiPw}" "created_at" "1468855106" "video_id" "2YhxfUejSxyEhTRkxdkd3Q" "modified_at" "1468855106" "creator_id" "3aOChHM5RMaGGz2cyjzUBQ" "conversation_id" "oYyu_NBS7CAMoebwh4nVSw" "message_id" "TXKwyOL1SROI_ENeFttiPw"
1468855106.197222 [0 127.0.0.1:62135] "eval" "\nlocal key = KEYS[1]\nlocal score = ARGV[1]\nlocal member = ARGV[2]\nlocal res = redis.call('zscore', key, member)\nif not res then\n    return redis.call('zadd', key, score, member)\nelse\n    return 0\nend\n" "1" "ZQ{conversation_archive}" "1468855106.196413" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.197261 [0 lua] "zscore" "ZQ{conversation_archive}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.197273 [0 lua] "zadd" "ZQ{conversation_archive}" "1468855106.196413" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.197347 [0 127.0.0.1:62135] "zadd" "CML{oYyu_NBS7CAMoebwh4nVSw}" "1468855106.195147" "TXKwyOL1SROI_ENeFttiPw"
1468855106.197389 [0 127.0.0.1:62135] "zadd" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "1468855106.196467" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.197455 [0 127.0.0.1:62135] "zadd" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106.196467" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.197788 [0 127.0.0.1:62135] "zrank" "CM{oYyu_NBS7CAMoebwh4nVSw}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.197939 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "500"
1468855106.198161 [0 127.0.0.1:62135] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "email" "phone" "push_badge" "invited"
1468855106.198783 [0 127.0.0.1:62135] "hgetall" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}"
1468855106.198932 [0 127.0.0.1:62135] "zrevrange" "ABEXPRBT{invite-hybrid}" "0" "-1"
1468855106.199120 [0 127.0.0.1:62135] "hmget" "ABEXP{4nnrrGcLRIKdNcVgtsHosA}" "invite_batch" "app_type" "name" "recruiting" "created_at" "modified_at" "recruiting_start" "exp_type" "platforms" "recruiting_end" "recruiting_location" "defaults" "active" "exp_id" "app_versions" "video_invite"
1468855106.199509 [0 127.0.0.1:62135] "zrevrange" "ABEXPCL{4nnrrGcLRIKdNcVgtsHosA}" "0" "-1"
1468855106.199672 [0 127.0.0.1:62135] "hmget" "ABEXPC{dqIVHNuoTdy8_ZSraeL7Xw}" "name" "cohort_id" "created_at" "modified_at" "exp_id" "overrides"
1468855106.199986 [0 127.0.0.1:62135] "hset" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}" "name" "IaLs5"
1468855106.200041 [0 127.0.0.1:62135] "hset" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}" "cohort" "variant"
1468855106.200101 [0 127.0.0.1:62135] "expire" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}" "600"
1468855106.200430 [0 127.0.0.1:62135] "hsetnx" "ConversationInviteIndex:35:u" "brother-t" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.200679 [0 127.0.0.1:62135] "hmset" "C{oYyu_NBS7CAMoebwh4nVSw}" "invite_id" "brother-t"
1468855106.201880 [0 127.0.0.1:62135] "hmset" "CIVTE2{oYyu_NBS7CAMoebwh4nVSw}" "sent_domain" "dev.videokiks.com" "sent_xps_name" "IaLs5" "sent_domain_group" "default" "sent_source" "invite opt in" "sent_video_invite" "1" "sent_batch" "suggested_client" "sent_quality_ordinal" "0" "sent_timestamp" "1468855106" "conversation_id" "oYyu_NBS7CAMoebwh4nVSw" "sent_contact_quality" "{\"C-NameLength\": 18, \"C-EmailCount\": 0, \"C-Quality\": 60, \"C-NameSingleWord\": false, \"C-SMSCount\": 10, \"C-Name\": \"Brother Test-ofsky\", \"C-ShortName\": false, \"C-HasNote\": false, \"C-RelatedPhonesCount\": 0, \"C-HasEmoticon\": false, \"C-WebsiteCount\": 0, \"C-FamilyName\": false, \"C-HasAvatar\": true, \"C-HasFacebook\": false, \"C-SameLastname\": true}" "sent_xps_cohort" "variant"
1468855106.202088 [0 127.0.0.1:62135] "expire" "CIVTE2{oYyu_NBS7CAMoebwh4nVSw}" "2592000"
1468855106.202312 [0 127.0.0.1:62135] "hmget" "UIVTR{4p8_3B94TZyqDJ0-VAUS8Q}" "user_id"
1468855106.203201 [0 127.0.0.1:62135] "hmget" "UIVTR{4p8_3B94TZyqDJ0-VAUS8Q}" "user_id"
1468855106.203890 [0 127.0.0.1:62135] "hgetall" "UAT{3aOChHM5RMaGGz2cyjzUBQ}"
1468855106.204284 [0 127.0.0.1:62135] "hgetall" "UEIBT{3aOChHM5RMaGGz2cyjzUBQ-invite-hybrid}"
1468855106.204967 [0 127.0.0.1:62135] "hmset" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "invited" "1"
1468855106.205231 [0 127.0.0.1:62135] "zrevrange" "UD{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "0"
1468855106.206887 [0 127.0.0.1:62135] "zrank" "CM{oYyu_NBS7CAMoebwh4nVSw}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.207182 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "-1"
1468855106.207492 [0 127.0.0.1:62135] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.207569 [0 127.0.0.1:62135] "hmget" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.209378 [0 127.0.0.1:62135] "hmget" "D{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.209902 [0 127.0.0.1:62135] "hmset" "D{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "app_type" "mp" "created_at" "1468855106" "platform_version" "16" "platform_type" "android" "secret" "sha256$d468a262063e$279394caf9b0d683186b778d85358f94eee08bf3abec9d4ac15e7050c23d07f0" "last_login" "1468855106" "timezone" "UTC" "flavor" "debug" "app_version" "1.1.0" "device_id" "Ae4jfZGoQ9Sw0OA9RTiN7Q"
1468855106.210511 [0 127.0.0.1:62135] "zrevrange" "ABEXPRBT{client}" "0" "-1"
1468855106.211619 [0 127.0.0.1:62135] "hmget" "D{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.212504 [0 127.0.0.1:62135] "hmget" "PC{+17855554005}" "code" "created_at" "contact_addr" "timeout" "recent_gateway" "user_list"
1468855106.212794 [0 127.0.0.1:62135] "incrby" "verifycode_generate_Ae4jfZGoQ9Sw0OA9RTiN7Q" "1"
1468855106.212888 [0 127.0.0.1:62135] "expire" "verifycode_generate_Ae4jfZGoQ9Sw0OA9RTiN7Q" "43200"
1468855106.213346 [0 127.0.0.1:62135] "del" "verifycode_generate_Ae4jfZGoQ9Sw0OA9RTiN7Q"
1468855106.213460 [0 127.0.0.1:62135] "del" "verifycode_submit_Ae4jfZGoQ9Sw0OA9RTiN7Q"
1468855106.213925 [0 127.0.0.1:62135] "hmset" "PC{+17855554005}" "created_at" "1468855106" "code" "3822" "timeout" "43200" "contact_addr" "+17855554005" "user_list" "[\"Ae4jfZGoQ9Sw0OA9RTiN7Q\"]"
1468855106.214508 [0 127.0.0.1:62135] "hget" "U:phone:17:u:" "+17855554005"
1468855106.214847 [0 127.0.0.1:62135] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.217165 [0 127.0.0.1:62135] "hmget" "D{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.218010 [0 127.0.0.1:62135] "hmget" "PC{+17855554005}" "code" "created_at" "contact_addr" "timeout" "recent_gateway" "user_list"
1468855106.218340 [0 127.0.0.1:62135] "incrby" "verifycode_submit_Ae4jfZGoQ9Sw0OA9RTiN7Q" "1"
1468855106.218394 [0 127.0.0.1:62135] "expire" "verifycode_submit_Ae4jfZGoQ9Sw0OA9RTiN7Q" "43200"
1468855106.218669 [0 127.0.0.1:62135] "del" "verifycode_generate_Ae4jfZGoQ9Sw0OA9RTiN7Q"
1468855106.218693 [0 127.0.0.1:62135] "del" "verifycode_submit_Ae4jfZGoQ9Sw0OA9RTiN7Q"
1468855106.219041 [0 127.0.0.1:62135] "hdel" "PC{+17855554005}" "code" "created_at" "contact_addr" "timeout" "recent_gateway" "user_list"
1468855106.219744 [0 127.0.0.1:62135] "hget" "U:phone:17:u:" "+17855554005"
1468855106.219990 [0 127.0.0.1:62135] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.220376 [0 127.0.0.1:62135] "zadd" "UD{4p8_3B94TZyqDJ0-VAUS8Q}" "1468855106.220233" "Ae4jfZGoQ9Sw0OA9RTiN7Q"
1468855106.220434 [0 127.0.0.1:62135] "hsetnx" "U:phone:17:u:" "+17855554005" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.220476 [0 127.0.0.1:62135] "hget" "U:phone:17:u:" "+17855554005"
1468855106.220691 [0 127.0.0.1:62135] "hmset" "D{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "ip" "" "user_id" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.220868 [0 127.0.0.1:62135] "zrem" "T-U{user_set_test_users}" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.222255 [0 127.0.0.1:62135] "hmget" "D{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.222736 [0 127.0.0.1:62135] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "email" "phone" "push_badge" "invited"
1468855106.223111 [0 127.0.0.1:62135] "del" "UIVTR{4p8_3B94TZyqDJ0-VAUS8Q}"
1468855106.223130 [0 127.0.0.1:62135] "hmset" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "James" "last_name" "Test-y" "email" "john+xihjfqbq@happybits.co" "signup_at" "1468855106"
1468855106.223511 [0 127.0.0.1:62135] "zcard" "UD{4p8_3B94TZyqDJ0-VAUS8Q}"
1468855106.224473 [0 127.0.0.1:62135] "zrange" "USR{Ae4jfZGoQ9Sw0OA9RTiN7Q}" "0" "-1"
1468855106.224653 [0 127.0.0.1:62135] "zrange" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "-1"
1468855106.224855 [0 127.0.0.1:62135] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.224913 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "3"
1468855106.225145 [0 127.0.0.1:62135] "del" "USR{Ae4jfZGoQ9Sw0OA9RTiN7Q}"
1468855106.225328 [0 127.0.0.1:62135] "zrange" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "-1"
1468855106.225486 [0 127.0.0.1:62135] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.225738 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "-1"
1468855106.225951 [0 127.0.0.1:62135] "zrevrange" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "0" "0"
1468855106.226129 [0 127.0.0.1:62135] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.226593 [0 127.0.0.1:62135] "hmget" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.226663 [0 127.0.0.1:62135] "zrevrange" "VCR{3aOChHM5RMaGGz2cyjzUBQ}" "0" "0" "WITHSCORES"
1468855106.226676 [0 127.0.0.1:62135] "zcard" "VCR{3aOChHM5RMaGGz2cyjzUBQ}"
1468855106.226683 [0 127.0.0.1:62135] "hget" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "signup_time"
1468855106.226708 [0 127.0.0.1:62135] "hgetall" "UAT{3aOChHM5RMaGGz2cyjzUBQ}"
1468855106.227122 [0 127.0.0.1:62135] "hgetall" "UEIBT{3aOChHM5RMaGGz2cyjzUBQ-invite-hybrid}"
1468855106.227275 [0 127.0.0.1:62135] "hincrby" "UAT{3aOChHM5RMaGGz2cyjzUBQ}" "count_friends" "1"
1468855106.227755 [0 127.0.0.1:62135] "zadd" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106.226975" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.227805 [0 127.0.0.1:62135] "zscore" "UML{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.227817 [0 127.0.0.1:62135] "hdel" "ConversationInviteIndex:35:u" "brother-t"
1468855106.227844 [0 127.0.0.1:62135] "hdel" "C{oYyu_NBS7CAMoebwh4nVSw}" "invite_id"
1468855106.228124 [0 127.0.0.1:62135] "zscore" "UBL{3aOChHM5RMaGGz2cyjzUBQ}" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.228340 [0 127.0.0.1:62135] "zrevrange" "ABEXPRBT{user_joined_push}" "0" "-1"
1468855106.228909 [0 127.0.0.1:62135] "zrevrange" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "0" "5"
1468855106.229206 [0 127.0.0.1:62135] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.229366 [0 127.0.0.1:62135] "get" "UDNDU{3aOChHM5RMaGGz2cyjzUBQ}"
1468855106.229753 [0 127.0.0.1:62135] "zrangebyscore" "PTB1{110ef9c637ba6c14a884c917c703819f}" "1468833506.229657" "+inf"
1468855106.230337 [0 127.0.0.1:62135] "set" "TPRT{bEdQGyj1TUaz7uFT3mNeJA.FJNq9}" "1468855106.230021"
1468855106.230409 [0 127.0.0.1:62135] "expire" "TPRT{bEdQGyj1TUaz7uFT3mNeJA.FJNq9}" "86400"
1468855106.230464 [0 127.0.0.1:62135] "publish" "#U{3aOChHM5RMaGGz2cyjzUBQ}" "{\"uid\": \"4p8_3B94TZyqDJ0-VAUS8Q\", \"cid\": \"oYyu_NBS7CAMoebwh4nVSw\", \"muted\": false, \"i\": \"FJNq9\", \"alert\": \"James Test-y just joined Marco Polo. Say hello!\", \"lname\": \"Test-y\", \"phone\": \"+17855554005\", \"fname\": \"James\", \"type\": \"is\"}"
1468855106.230947 [0 127.0.0.1:62135] "incrby" "UPBD{3aOChHM5RMaGGz2cyjzUBQ-17000}" "1"
1468855106.230979 [0 127.0.0.1:62135] "expire" "UPBD{3aOChHM5RMaGGz2cyjzUBQ-17000}" "86400"
1468855106.231015 [0 127.0.0.1:62135] "incrby" "UPBD{3aOChHM5RMaGGz2cyjzUBQ-408015}" "1"
1468855106.231041 [0 127.0.0.1:62135] "expire" "UPBD{3aOChHM5RMaGGz2cyjzUBQ-408015}" "3600"
1468855106.231345 [0 127.0.0.1:62135] "zadd" "PTAO{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106.231281" "3hJI6eYvSSOSS57pI1DA6w"
1468855106.231497 [0 127.0.0.1:62135] "expire" "PTAO{3aOChHM5RMaGGz2cyjzUBQ}" "61"
1468855106.231677 [0 127.0.0.1:62135] "zadd" "PTR{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106.231598" "3hJI6eYvSSOSS57pI1DA6w"
1468855106.231826 [0 127.0.0.1:62135] "expire" "PTR{3aOChHM5RMaGGz2cyjzUBQ}" "601"
1468855106.232179 [0 127.0.0.1:62135] "zrange" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "-1"
1468855106.232354 [0 127.0.0.1:62135] "hmget" "CIVTE2{oYyu_NBS7CAMoebwh4nVSw}" "sent_xps_name" "sent_source" "sent_video_invite" "clicked_domain_group" "clicked_video_invite" "clicked_source" "sent_reinvite_ordinal" "clicked_timestamp" "clicked_quality_ordinal" "sent_domain_group" "clicked_contact_quality" "sent_batch" "clicked_reinvite_ordinal" "sent_contact_quality" "clicked_domain" "clicked_xps_cohort" "clicked_batch" "conversation_id" "clicked_xps_name" "sent_domain" "sent_quality_ordinal" "sent_timestamp" "sent_xps_cohort"
1468855106.232756 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "-1"
1468855106.232923 [0 127.0.0.1:62135] "hmget" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.233246 [0 127.0.0.1:62135] "zrevrange" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "0" "0"
1468855106.233458 [0 127.0.0.1:62135] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.233937 [0 127.0.0.1:62135] "hgetall" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}"
1468855106.234838 [0 127.0.0.1:62135] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "-1"
1468855106.235073 [0 127.0.0.1:62135] "hmget" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.235448 [0 127.0.0.1:62135] "zrevrange" "UD{3aOChHM5RMaGGz2cyjzUBQ}" "0" "0"
1468855106.235637 [0 127.0.0.1:62135] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.236174 [0 127.0.0.1:62135] "hgetall" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}"
1468855106.236284 [0 127.0.0.1:62135] "hgetall" "UEIBT{4p8_3B94TZyqDJ0-VAUS8Q-invite-hybrid}"
1468855106.236490 [0 127.0.0.1:62135] "del" "SSMS{+17855554005}"
1468855106.236782 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_opens" "1"
1468855106.236824 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "activity_alerts" "on"
1468855106.236885 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_records" "0"
1468855106.236947 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_groups" "0"
1468855106.236989 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "current_orbit" "signup"
1468855106.237040 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "signup_tracked" "1"
1468855106.237083 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_friends" "0"
1468855106.237137 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "invited" "1"
1468855106.237182 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_recordDays" "0"
1468855106.237224 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_newplays" "0"
1468855106.237257 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "signup_time" "1468855106"
1468855106.237296 [0 127.0.0.1:62135] "hset" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "signup_tz" "UTC"
1468855106.237873 [0 127.0.0.1:62135] "zrange" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "-1"
1468855106.238038 [0 127.0.0.1:62135] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.238286 [0 127.0.0.1:62135] "hincrby" "UAT{4p8_3B94TZyqDJ0-VAUS8Q}" "count_friends" "1"
1468855106.238909 [0 127.0.0.1:62135] "hdel" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "invited"
1468855106.240920 [0 127.0.0.1:62135] "hmget" "V{2YhxfUejSxyEhTRkxdkd3Q}" "status" "container" "deleted" "created_at" "video_id" "video_type" "upload_pending" "upload_server" "upload_service" "path_secret" "user_id" "local_upload_complete"
1468855106.241296 [0 127.0.0.1:62136] "hmget" "D{bEdQGyj1TUaz7uFT3mNeJA}" "flavor" "locale" "ip" "push_token" "timezone" "voip_token" "experiment_active" "app_build" "manufacturer" "cohort" "app_type" "platform_version" "secret" "experiment" "last_login" "app_version" "user_id" "device_id" "created_at" "platform_type" "carrier" "model_name"
1468855106.241437 [0 127.0.0.1:62136] "hmget" "U{3aOChHM5RMaGGz2cyjzUBQ}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.242011 [0 127.0.0.1:62136] "hmget" "M{TXKwyOL1SROI_ENeFttiPw}" "by" "deleted" "text" "created_at" "video_id" "modified_at" "emotes" "event" "creator_id" "conversation_id" "message_id"
1468855106.242536 [0 127.0.0.1:62136] "hget" "U:phone:17:u:" "+17855554005"
1468855106.242694 [0 127.0.0.1:62136] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "phone" "push_badge" "email"
1468855106.243063 [0 127.0.0.1:62136] "zrange" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "0" "-1"
1468855106.243094 [0 127.0.0.1:62136] "zrange" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "0" "-1"
1468855106.243301 [0 127.0.0.1:62136] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.243367 [0 127.0.0.1:62136] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "3"
1468855106.243654 [0 127.0.0.1:62136] "hmget" "C{oYyu_NBS7CAMoebwh4nVSw}" "group" "title" "modified_at" "icon_id" "invite_id" "archive_mark" "conversation_id" "created_at"
1468855106.243907 [0 127.0.0.1:62136] "hdel" "C{oYyu_NBS7CAMoebwh4nVSw}" "title"
1468855106.244141 [0 127.0.0.1:62136] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "-1"
1468855106.244166 [0 127.0.0.1:62136] "zscore" "UBL{4p8_3B94TZyqDJ0-VAUS8Q}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.244176 [0 127.0.0.1:62136] "zscore" "UBL{3aOChHM5RMaGGz2cyjzUBQ}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.244185 [0 127.0.0.1:62136] "hget" "UCRM{4p8_3B94TZyqDJ0-VAUS8Q}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244199 [0 127.0.0.1:62136] "hget" "UCRM{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244765 [0 127.0.0.1:62136] "zadd" "CM{oYyu_NBS7CAMoebwh4nVSw}" "1468855106.244314" "4p8_3B94TZyqDJ0-VAUS8Q"
1468855106.244813 [0 127.0.0.1:62136] "zrem" "UCR{4p8_3B94TZyqDJ0-VAUS8Q}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244830 [0 127.0.0.1:62136] "hdel" "UCRM{4p8_3B94TZyqDJ0-VAUS8Q}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244840 [0 127.0.0.1:62136] "zadd" "CM{oYyu_NBS7CAMoebwh4nVSw}" "1468855106.244314" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.244895 [0 127.0.0.1:62136] "zrem" "UCR{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244911 [0 127.0.0.1:62136] "hdel" "UCRM{3aOChHM5RMaGGz2cyjzUBQ}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244932 [0 127.0.0.1:62136] "zadd" "UC{4p8_3B94TZyqDJ0-VAUS8Q}" "1468855106.244617" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.244982 [0 127.0.0.1:62136] "zadd" "UC{3aOChHM5RMaGGz2cyjzUBQ}" "1468855106.244617" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.245380 [0 127.0.0.1:62136] "hmget" "V{2YhxfUejSxyEhTRkxdkd3Q}" "status" "container" "deleted" "created_at" "video_id" "video_type" "upload_pending" "upload_server" "upload_service" "path_secret" "user_id" "local_upload_complete"
1468855106.246106 [0 127.0.0.1:62136] "zrank" "CM{oYyu_NBS7CAMoebwh4nVSw}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.246332 [0 127.0.0.1:62136] "hdel" "M{TXKwyOL1SROI_ENeFttiPw}" "text" "event"
1468855106.246364 [0 127.0.0.1:62136] "eval" "\nlocal key = KEYS[1]\nlocal score = ARGV[1]\nlocal member = ARGV[2]\nlocal res = redis.call('zscore', key, member)\nif not res then\n    return redis.call('zadd', key, score, member)\nelse\n    return 0\nend\n" "1" "ZQ{conversation_archive}" "1468855106.246236" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.246557 [0 lua] "zscore" "ZQ{conversation_archive}" "oYyu_NBS7CAMoebwh4nVSw"
1468855106.246581 [0 127.0.0.1:62136] "zadd" "CML{oYyu_NBS7CAMoebwh4nVSw}" "1468855106" "TXKwyOL1SROI_ENeFttiPw"
1468855106.246806 [0 127.0.0.1:62136] "zrank" "CM{oYyu_NBS7CAMoebwh4nVSw}" "3aOChHM5RMaGGz2cyjzUBQ"
1468855106.246966 [0 127.0.0.1:62136] "zrange" "CM{oYyu_NBS7CAMoebwh4nVSw}" "0" "500"
1468855106.247188 [0 127.0.0.1:62136] "hmget" "U{4p8_3B94TZyqDJ0-VAUS8Q}" "first_name" "last_name" "user_id" "last_active" "icon_id" "fbid" "signup_at" "email" "phone" "push_badge" "invited"
Error: Server closed the connection
yinqiwen commented 8 years ago

i just reproduce this issue in my mac now, seems easy to reproduce.

yinqiwen commented 8 years ago

just found the root cause of this issue, in function GetMultiStringCachehttps://github.com/yinqiwen/ardb/blob/ee4ede41452c9d002e39a2b6d35f4ebd04c1ab09/src/db/rocksdb/rocksdb_engine.cpp#L255-L265 , when caller required more than DEFAULT_ROCKS_LOCAL_MULTI_CACHE_SIZE(10), this function only return an array with 10 size.

72squared commented 8 years ago

That seems to fix the segfault issue, at least at first pass. will do a bit more thorough testing. Now that I am able to run my unit tests I am discovering other subtle differences between ardb and the redis reference implementation. I'll open those as separate issues.

72squared commented 8 years ago

made it through several full runs of my application's test suite without segfaults. Progress! Before I couldn't even make it through 1 run of a subset of my tests without a crash. Closing this issue. I'll re-open if I see additional issues in this area.