search

Tokutek / mongo

TokuMX is a high-performance, concurrent, compressing, drop-in replacement engine for MongoDB | Issue tracker: https://tokutek.atlassian.net/browse/MX/ |
http://www.tokutek.com/products/tokumx-for-mongodb/
704 stars 97 forks source link

tokumx crash #1219

Open m0n5t3r opened 9 years ago

m0n5t3r commented 9 years ago

We have a replica set with 2 replicas + arbiter; this morning one of the replicas started crashing (it's gotten to the point where it's immediately after start-up, but initially it did work for a few minutes between crashes)

incoming crash logs:

first crash, the machine was primary at the moment:

Checksum failure while reading node partition in file somecollection.tokumx.
Fri Mar 27 05:01:28.556  
Fri Mar 27 05:01:28.556 ================================================================================
Fri Mar 27 05:01:28.556  Fatal error detected
Fri Mar 27 05:01:28.556 ================================================================================
Fri Mar 27 05:01:28.556  
Fri Mar 27 05:01:28.556 About to gather debugging information, please include all of the following along
Fri Mar 27 05:01:28.556 with logs from other servers in the cluster in a bug report.
Fri Mar 27 05:01:28.556  
Fri Mar 27 05:01:28.556 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.556 Version info:
Fri Mar 27 05:01:28.556  
Fri Mar 27 05:01:28.556 tokumxVersion: 2.0.0
Fri Mar 27 05:01:28.556 gitVersion: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 05:01:28.556 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 05:01:28.556 sysInfo: Linux 3eb534264b3c 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_54
Fri Mar 27 05:01:28.556 loaderFlags:        
Fri Mar 27 05:01:28.556 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local
-typedefs -fno-builtin-memcmp -O3
Fri Mar 27 05:01:28.556 debug: false
Fri Mar 27 05:01:28.556  
Fri Mar 27 05:01:28.556 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.556 Simple stacktrace:
Fri Mar 27 05:01:28.556  
0xb0d7f6 0x876e44 0x879162 0x6d85f1 0x7f7da6933c30 0x7f7da6933bb9 0x7f7da6936fc8 0x7f7da81f05a2 0x7f7da81f08e3 0x7f7da81eaeb7 0x7f7da81eb010 0x7f7da820459f 0x7f7da8197727 0x7f7da81983d2 0x7f7da846f182 0x7f7da69f7efd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xb0d7f6]
 /usr/bin/mongod() [0x876e44]
 /usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x32) [0x879162]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x1f1) [0x6d85f1]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7f7da6933c30]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f7da6933bb9]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f7da6936fc8]
 /usr/lib/tokumx/libtokufractaltree.so(+0x955a2) [0x7f7da81f05a2]
 /usr/lib/tokumx/libtokufractaltree.so(+0x958e3) [0x7f7da81f08e3]
 /usr/lib/tokumx/libtokufractaltree.so(+0x8feb7) [0x7f7da81eaeb7]
 /usr/lib/tokumx/libtokufractaltree.so(+0x90010) [0x7f7da81eb010]
 /usr/lib/tokumx/libtokufractaltree.so(+0xa959f) [0x7f7da820459f]
 /usr/lib/tokumx/libtokufractaltree.so(+0x3c727) [0x7f7da8197727]
 /usr/lib/tokumx/libtokufractaltree.so(+0x3d3d2) [0x7f7da81983d2]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f7da846f182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7da69f7efd]
Fri Mar 27 05:01:28.557  
Fri Mar 27 05:01:28.557 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.557 TokuKV engine backtrace:
Fri Mar 27 05:01:28.557  
Fri Mar 27 05:01:28.557 Backtrace: (Note: toku_do_assert=0x0x7f7da7f53ce0)

Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokuportability.so(_Z27db_env_do_backtrace_errfuncPFvPKviPKczES0_+0x1f) [0x7f7da7f555bf]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x149cd) [0x7f7da816f9cd]
Fri Mar 27 05:01:28.558 /usr/bin/mongod() [0x876f5d]
Fri Mar 27 05:01:28.558 /usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x37) [0x879167]
Fri Mar 27 05:01:28.558 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x1f1) [0x6d85f1]
Fri Mar 27 05:01:28.558 /lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7f7da6933c30]
Fri Mar 27 05:01:28.558 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f7da6933bb9]
Fri Mar 27 05:01:28.558 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f7da6936fc8]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x955a2) [0x7f7da81f05a2]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x958e3) [0x7f7da81f08e3]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x8feb7) [0x7f7da81eaeb7]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x90010) [0x7f7da81eb010]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0xa959f) [0x7f7da820459f]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x3c727) [0x7f7da8197727]
Fri Mar 27 05:01:28.558 /usr/lib/tokumx/libtokufractaltree.so(+0x3d3d2) [0x7f7da81983d2]
Fri Mar 27 05:01:28.558 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f7da846f182]
Fri Mar 27 05:01:28.558 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7da69f7efd]
Fri Mar 27 05:01:28.558 BUILD_ID = 0
Fri Mar 27 05:01:28.558 time of environment creation: Mon Jan 12 19:57:28 2015
Fri Mar 27 05:01:28.558 time of engine startup: Thu Mar 26 10:52:21 2015
Fri Mar 27 05:01:28.558 time now: Fri Mar 27 05:01:28 2015
Fri Mar 27 05:01:28.558 db opens: 121
Fri Mar 27 05:01:28.558 db closes: 2
Fri Mar 27 05:01:28.558 num open dbs now: 119
Fri Mar 27 05:01:28.558 max open dbs: 119
Fri Mar 27 05:01:28.558 period, in ms, that recovery log is automatically fsynced: 100
Fri Mar 27 05:01:28.558 dictionary inserts: 2
Fri Mar 27 05:01:28.558 dictionary inserts fail: 0
Fri Mar 27 05:01:28.558 dictionary deletes: 2
Fri Mar 27 05:01:28.558 dictionary deletes fail: 0
Fri Mar 27 05:01:28.558 dictionary updates: 2979695
Fri Mar 27 05:01:28.558 dictionary updates fail: 0
Fri Mar 27 05:01:28.558 dictionary broadcast updates: 0
Fri Mar 27 05:01:28.558 dictionary broadcast updates fail: 0
Fri Mar 27 05:01:28.558 dictionary multi inserts: 3747637
Fri Mar 27 05:01:28.558 dictionary multi inserts fail: 0
Fri Mar 27 05:01:28.558 dictionary multi deletes: 2292844
Fri Mar 27 05:01:28.558 dictionary multi deletes fail: 0
Fri Mar 27 05:01:28.558 dictionary updates multi: 1916072
Fri Mar 27 05:01:28.558 dictionary updates multi fail: 0
Fri Mar 27 05:01:28.558 le: max committed xr: 2
Fri Mar 27 05:01:28.558 le: max provisional xr: 2
Fri Mar 27 05:01:28.558 le: expanded: 0
Fri Mar 27 05:01:28.558 le: max memsize: 3178719
Fri Mar 27 05:01:28.558 le: size of leafentries before garbage collection (during message application): 26085103964
Fri Mar 27 05:01:28.558 le: size of leafentries after garbage collection (during message application): 15976507287
Fri Mar 27 05:01:28.558 le: size of leafentries before garbage collection (outside message application): 0
Fri Mar 27 05:01:28.558 le: size of leafentries after garbage collection (outside message application): 0
Fri Mar 27 05:01:28.558 checkpoint: period: 60
Fri Mar 27 05:01:28.558 checkpoint: footprint: 0
Fri Mar 27 05:01:28.558 checkpoint: last checkpoint began : Fri Mar 27 05:01:21 2015
Fri Mar 27 05:01:28.559 checkpoint: last complete checkpoint began : Fri Mar 27 05:01:21 2015
Fri Mar 27 05:01:28.559 checkpoint: last complete checkpoint ended: Fri Mar 27 05:01:24 2015
Fri Mar 27 05:01:28.559 checkpoint: time spent during checkpoint (begin and end phases): 3414
Fri Mar 27 05:01:28.559 checkpoint: time spent during last checkpoint (begin and end phases): 3
Fri Mar 27 05:01:28.559 checkpoint: last complete checkpoint LSN: 1407606434
Fri Mar 27 05:01:28.559 checkpoint: checkpoints taken : 1091
Fri Mar 27 05:01:28.559 checkpoint: checkpoints failed: 0
Fri Mar 27 05:01:28.559 checkpoint: waiters now: 0
Fri Mar 27 05:01:28.559 checkpoint: waiters max: 0
Fri Mar 27 05:01:28.559 checkpoint: non-checkpoint client wait on mo lock: 20
Fri Mar 27 05:01:28.559 checkpoint: non-checkpoint client wait on cs lock: 0
Fri Mar 27 05:01:28.559 checkpoint: checkpoint begin time: 1560528
Fri Mar 27 05:01:28.559 checkpoint: long checkpoint begin time: 0
Fri Mar 27 05:01:28.559 checkpoint: long checkpoint begin count: 0
Fri Mar 27 05:01:28.559 cachetable: miss: 6109
Fri Mar 27 05:01:28.559 cachetable: miss time: 12997796
Fri Mar 27 05:01:28.559 cachetable: prefetches: 11803
Fri Mar 27 05:01:28.559 cachetable: size current: 16792927690
Fri Mar 27 05:01:28.559 cachetable: size limit: 17334624256
Fri Mar 27 05:01:28.559 cachetable: size writing: 0
Fri Mar 27 05:01:28.559 cachetable: size nonleaf: 787929143
Fri Mar 27 05:01:28.559 cachetable: size leaf: 16004997923
Fri Mar 27 05:01:28.559 cachetable: size rollback: 624
Fri Mar 27 05:01:28.559 cachetable: size cachepressure: 780495138
Fri Mar 27 05:01:28.559 cachetable: size currently cloned data for checkpoint: 0
Fri Mar 27 05:01:28.559 cachetable: evictions: 1089
Fri Mar 27 05:01:28.559 cachetable: cleaner executions: 35889
Fri Mar 27 05:01:28.559 cachetable: cleaner period: 2
Fri Mar 27 05:01:28.559 cachetable: cleaner iterations: 5
Fri Mar 27 05:01:28.559 cachetable: number of waits on cache pressure: 0
Fri Mar 27 05:01:28.559 cachetable: time waiting on cache pressure: 0
Fri Mar 27 05:01:28.559 cachetable: number of long waits on cache pressure: 0
Fri Mar 27 05:01:28.559 cachetable: long time waiting on cache pressure: 0
Fri Mar 27 05:01:28.559 locktree: memory size: 0
Fri Mar 27 05:01:28.559 locktree: memory size limit: 1679775334
Fri Mar 27 05:01:28.559 locktree: number of times lock escalation ran: 0
Fri Mar 27 05:01:28.559 locktree: time spent running escalation (seconds): 0.000000
Fri Mar 27 05:01:28.559 locktree: latest post-escalation memory size: 0
Fri Mar 27 05:01:28.559 locktree: number of locktrees open now: 121
Fri Mar 27 05:01:28.559 locktree: number of pending lock requests: 0
Fri Mar 27 05:01:28.559 locktree: number of locktrees eligible for the STO: 0
Fri Mar 27 05:01:28.559 locktree: number of times a locktree ended the STO early: 1
Fri Mar 27 05:01:28.559 locktree: time spent ending the STO early (seconds): 0.000004
Fri Mar 27 05:01:28.559 locktree: number of wait locks: 0
Fri Mar 27 05:01:28.559 locktree: time waiting for locks: 0
Fri Mar 27 05:01:28.559 locktree: number of long wait locks: 0
Fri Mar 27 05:01:28.559 locktree: long time waiting for locks: 0
Fri Mar 27 05:01:28.559 locktree: number of lock timeouts: 0
Fri Mar 27 05:01:28.559 locktree: number of waits on lock escalation: 0
Fri Mar 27 05:01:28.559 locktree: time waiting on lock escalation: 0
Fri Mar 27 05:01:28.559 locktree: number of long waits on lock escalation: 0
Fri Mar 27 05:01:28.559 locktree: long time waiting on lock escalation: 0
Fri Mar 27 05:01:28.559 ft: dictionary updates: 2975569
Fri Mar 27 05:01:28.559 ft: dictionary broadcast updates: 0
Fri Mar 27 05:01:28.559 ft: descriptor set: 2
Fri Mar 27 05:01:28.559 ft: messages ignored by leaf due to msn: 1246
Fri Mar 27 05:01:28.559 ft: total search retries due to TRY_AGAIN: 14904
Fri Mar 27 05:01:28.559 ft: searches requiring more tries than the height of the tree: 0
Fri Mar 27 05:01:28.559 ft: searches requiring more tries than the height of the tree plus three: 0
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (not for checkpoint): 18
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 7653888
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 34615817
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.001496
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (not for checkpoint): 25
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 11703296
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 44346712
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.002372
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (for checkpoint): 29549
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (for checkpoint) (bytes): 10838546432
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 47318770696
Fri Mar 27 05:01:28.559 ft: leaf nodes flushed to disk (for checkpoint) (seconds): 2.881090
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (for checkpoint): 38566
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 13381863424
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 51681468120
Fri Mar 27 05:01:28.559 ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 2.894310
Fri Mar 27 05:01:28.559 ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 05:01:28.559 ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 05:01:28.559 ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
Fri Mar 27 05:01:28.559 ft: nonleaf node partial evictions: 0
Fri Mar 27 05:01:28.559 ft: nonleaf node partial evictions (bytes): 0
Fri Mar 27 05:01:28.559 ft: leaf node partial evictions: 633965
Fri Mar 27 05:01:28.559 ft: leaf node partial evictions (bytes): 51057658086
Fri Mar 27 05:01:28.559 ft: leaf node full evictions: 901
Fri Mar 27 05:01:28.559 ft: leaf node full evictions (bytes): 46997044
Fri Mar 27 05:01:28.559 ft: nonleaf node full evictions: 188
Fri Mar 27 05:01:28.559 ft: nonleaf node full evictions (bytes): 73773690
Fri Mar 27 05:01:28.559 ft: leaf nodes created: 1555
Fri Mar 27 05:01:28.559 ft: nonleaf nodes created: 159
Fri Mar 27 05:01:28.559 ft: leaf nodes destroyed: 0
Fri Mar 27 05:01:28.559 ft: nonleaf nodes destroyed: 0
Fri Mar 27 05:01:28.559 ft: bytes of messages injected at root (all trees): 951158981
Fri Mar 27 05:01:28.559 ft: bytes of messages flushed from h1 nodes to leaves: 919467528
Fri Mar 27 05:01:28.559 ft: bytes of messages currently in trees (estimate): 31691453
Fri Mar 27 05:01:28.559 ft: messages injected at root: 6161984
Fri Mar 27 05:01:28.559 ft: broadcast messages injected at root: 0
Fri Mar 27 05:01:28.559 ft: basements decompressed as a target of a query: 1
Fri Mar 27 05:01:28.559 ft: basements decompressed for prelocked range: 0
Fri Mar 27 05:01:28.559 ft: basements decompressed for prefetch: 0
Fri Mar 27 05:01:28.559 ft: basements decompressed for write: 0
Fri Mar 27 05:01:28.559 ft: buffers decompressed as a target of a query: 0
Fri Mar 27 05:01:28.559 ft: buffers decompressed for prelocked range: 0
Fri Mar 27 05:01:28.559 ft: buffers decompressed for prefetch: 0
Fri Mar 27 05:01:28.559 ft: buffers decompressed for write: 0
Fri Mar 27 05:01:28.559 ft: pivots fetched for query: 1483
Fri Mar 27 05:01:28.559 ft: pivots fetched for query (bytes): 48594944
Fri Mar 27 05:01:28.559 ft: pivots fetched for query (seconds): 0.006302
Fri Mar 27 05:01:28.559 ft: pivots fetched for prefetch: 11803
Fri Mar 27 05:01:28.559 ft: pivots fetched for prefetch (bytes): 386760192
Fri Mar 27 05:01:28.559 ft: pivots fetched for prefetch (seconds): 0.075030
Fri Mar 27 05:01:28.559 ft: pivots fetched for write: 323
Fri Mar 27 05:01:28.559 ft: pivots fetched for write (bytes): 5231616
Fri Mar 27 05:01:28.559 ft: pivots fetched for write (seconds): 0.020985
Fri Mar 27 05:01:28.559 ft: basements fetched as a target of a query: 1486
Fri Mar 27 05:01:28.559 ft: basements fetched as a target of a query (bytes): 12849664
Fri Mar 27 05:01:28.559 ft: basements fetched as a target of a query (seconds): 0.000986
Fri Mar 27 05:01:28.559 ft: basements fetched for prelocked range: 79143
Fri Mar 27 05:01:28.559 ft: basements fetched for prelocked range (bytes): 1001997824
Fri Mar 27 05:01:28.559 ft: basements fetched for prelocked range (seconds): 0.119409
Fri Mar 27 05:01:28.559 ft: basements fetched for prefetch: 612045
Fri Mar 27 05:01:28.559 ft: basements fetched for prefetch (bytes): 7546894336
Fri Mar 27 05:01:28.559 ft: basements fetched for prefetch (seconds): 0.910073
Fri Mar 27 05:01:28.559 ft: basements fetched for write: 75872
Fri Mar 27 05:01:28.559 ft: basements fetched for write (bytes): 1524901376
Fri Mar 27 05:01:28.559 ft: basements fetched for write (seconds): 3.121403
Fri Mar 27 05:01:28.559 ft: buffers fetched as a target of a query: 0
Fri Mar 27 05:01:28.559 ft: buffers fetched as a target of a query (bytes): 0
Fri Mar 27 05:01:28.559 ft: buffers fetched as a target of a query (seconds): 0.000000
Fri Mar 27 05:01:28.559 ft: buffers fetched for prelocked range: 0
Fri Mar 27 05:01:28.559 ft: buffers fetched for prelocked range (bytes): 0
Fri Mar 27 05:01:28.559 ft: buffers fetched for prelocked range (seconds): 0.000000
Fri Mar 27 05:01:28.559 ft: buffers fetched for prefetch: 0
Fri Mar 27 05:01:28.559 ft: buffers fetched for prefetch (bytes): 0
Fri Mar 27 05:01:28.559 ft: buffers fetched for prefetch (seconds): 0.000000
Fri Mar 27 05:01:28.559 ft: buffers fetched for write: 7953
Fri Mar 27 05:01:28.559 ft: buffers fetched for write (bytes): 141093888
Fri Mar 27 05:01:28.559 ft: buffers fetched for write (seconds): 0.023609
Fri Mar 27 05:01:28.559 ft: leaf compression to memory (seconds): 556.846440
Fri Mar 27 05:01:28.559 ft: leaf serialization to memory (seconds): 14.886162
Fri Mar 27 05:01:28.559 ft: leaf decompression to memory (seconds): 77.027153
Fri Mar 27 05:01:28.559 ft: leaf deserialization to memory (seconds): 8.661888
Fri Mar 27 05:01:28.559 ft: nonleaf compression to memory (seconds): 682.773049
Fri Mar 27 05:01:28.559 ft: nonleaf serialization to memory (seconds): 20.804160
Fri Mar 27 05:01:28.559 ft: nonleaf decompression to memory (seconds): 0.941205
Fri Mar 27 05:01:28.559 ft: nonleaf deserialization to memory (seconds): 0.362455
Fri Mar 27 05:01:28.559 ft: promotion: roots split: 6
Fri Mar 27 05:01:28.559 ft: promotion: leaf roots injected into: 195244
Fri Mar 27 05:01:28.559 ft: promotion: h1 roots injected into: 111015
Fri Mar 27 05:01:28.559 ft: promotion: injections at depth 0: 980566
Fri Mar 27 05:01:28.559 ft: promotion: injections at depth 1: 4724723
Fri Mar 27 05:01:28.559 ft: promotion: injections at depth 2: 905805
Fri Mar 27 05:01:28.559 ft: promotion: injections at depth 3: 2093865
Fri Mar 27 05:01:28.559 ft: promotion: injections lower than depth 3: 1259278
Fri Mar 27 05:01:28.559 ft: promotion: stopped because of a nonempty buffer: 6008292
Fri Mar 27 05:01:28.559 ft: promotion: stopped at height 1: 852
Fri Mar 27 05:01:28.560 ft: promotion: stopped because the child was locked or not at all in memory: 121
Fri Mar 27 05:01:28.560 ft: promotion: stopped because the child was not fully in memory: 0
Fri Mar 27 05:01:28.560 ft: promotion: stopped anyway, after locking the child: 614
Fri Mar 27 05:01:28.560 ft: basement nodes deserialized with fixed-keysize: 744115
Fri Mar 27 05:01:28.560 ft: basement nodes deserialized with variable-keysize: 24444
Fri Mar 27 05:01:28.560 ft: promotion: succeeded in using the rightmost leaf shortcut: 2284667
Fri Mar 27 05:01:28.560 ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 309
Fri Mar 27 05:01:28.560 ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 723292
Fri Mar 27 05:01:28.560 ft flusher: total nodes potentially flushed by cleaner thread: 3444
Fri Mar 27 05:01:28.560 ft flusher: height-one nodes flushed by cleaner thread: 2862
Fri Mar 27 05:01:28.560 ft flusher: height-greater-than-one nodes flushed by cleaner thread: 582
Fri Mar 27 05:01:28.560 ft flusher: nodes cleaned which had empty buffers: 0
Fri Mar 27 05:01:28.560 ft flusher: nodes dirtied by cleaner thread: 6744
Fri Mar 27 05:01:28.560 ft flusher: max bytes in a buffer flushed by cleaner thread: 3812200
Fri Mar 27 05:01:28.560 ft flusher: min bytes in a buffer flushed by cleaner thread: 58
Fri Mar 27 05:01:28.560 ft flusher: total bytes in buffers flushed by cleaner thread: 841217554
Fri Mar 27 05:01:28.560 ft flusher: max workdone in a buffer flushed by cleaner thread: 1243458
Fri Mar 27 05:01:28.560 ft flusher: min workdone in a buffer flushed by cleaner thread: 0
Fri Mar 27 05:01:28.560 ft flusher: total workdone in buffers flushed by cleaner thread: 15229892
Fri Mar 27 05:01:28.560 ft flusher: times cleaner thread tries to merge a leaf: 24
Fri Mar 27 05:01:28.560 ft flusher: cleaner thread leaf merges in progress: 0
Fri Mar 27 05:01:28.560 ft flusher: cleaner thread leaf merges successful: 23
Fri Mar 27 05:01:28.560 ft flusher: nodes dirtied by cleaner thread leaf merges: 23
Fri Mar 27 05:01:28.560 ft flusher: total number of flushes done by flusher threads or cleaner threads: 3946
Fri Mar 27 05:01:28.560 ft flusher: number of in memory flushes: 3946
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that read something off disk: 0
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered another flush in child: 67
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered 1 cascading flush: 64
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered 2 cascading flushes: 3
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered 3 cascading flushes: 0
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered 4 cascading flushes: 0
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered 5 cascading flushes: 0
Fri Mar 27 05:01:28.560 ft flusher: number of flushes that triggered over 5 cascading flushes: 0
Fri Mar 27 05:01:28.560 ft flusher: leaf node splits: 1553
Fri Mar 27 05:01:28.560 ft flusher: nonleaf node splits: 153
Fri Mar 27 05:01:28.560 ft flusher: leaf node merges: 23
Fri Mar 27 05:01:28.560 ft flusher: nonleaf node merges: 0
Fri Mar 27 05:01:28.560 ft flusher: leaf node balances: 0
Fri Mar 27 05:01:28.560 hot: operations ever started: 0
Fri Mar 27 05:01:28.560 hot: operations successfully completed: 0
Fri Mar 27 05:01:28.560 hot: operations aborted: 0
Fri Mar 27 05:01:28.560 hot: max number of flushes from root ever required to optimize a tree: 0
Fri Mar 27 05:01:28.560 txn: begin: 5898911
Fri Mar 27 05:01:28.560 txn: begin read only: 2177
Fri Mar 27 05:01:28.560 txn: successful commits: 5899995
Fri Mar 27 05:01:28.560 txn: aborts: 1093
Fri Mar 27 05:01:28.560 logger: next LSN: 1407606798
Fri Mar 27 05:01:28.560 logger: writes: 255197
Fri Mar 27 05:01:28.560 logger: writes (bytes): 4572456583
Fri Mar 27 05:01:28.560 logger: writes (uncompressed bytes): 4572456583
Fri Mar 27 05:01:28.560 logger: writes (seconds): 3.420198
Fri Mar 27 05:01:28.560 logger: number of long logger write operations: 0
Fri Mar 27 05:01:28.560 indexer: number of indexers successfully created: 0
Fri Mar 27 05:01:28.560 indexer: number of calls to toku_indexer_create_indexer() that failed: 0
Fri Mar 27 05:01:28.560 indexer: number of calls to indexer->build() succeeded: 0
Fri Mar 27 05:01:28.560 indexer: number of calls to indexer->build() failed: 0
Fri Mar 27 05:01:28.560 indexer: number of calls to indexer->close() that succeeded: 0
Fri Mar 27 05:01:28.560 indexer: number of calls to indexer->close() that failed: 0
Fri Mar 27 05:01:28.560 indexer: number of calls to indexer->abort(): 0
Fri Mar 27 05:01:28.560 indexer: number of indexers currently in existence: 0
Fri Mar 27 05:01:28.560 indexer: max number of indexers that ever existed simultaneously: 0
Fri Mar 27 05:01:28.560 loader: number of loaders successfully created: 0
Fri Mar 27 05:01:28.560 loader: number of calls to toku_loader_create_loader() that failed: 0
Fri Mar 27 05:01:28.560 loader: number of calls to loader->put() succeeded: 0
Fri Mar 27 05:01:28.560 loader: number of calls to loader->put() failed: 0
Fri Mar 27 05:01:28.560 loader: number of calls to loader->close() that succeeded: 0
Fri Mar 27 05:01:28.560 loader: number of calls to loader->close() that failed: 0
Fri Mar 27 05:01:28.560 loader: number of calls to loader->abort(): 0
Fri Mar 27 05:01:28.560 loader: number of loaders currently in existence: 0
Fri Mar 27 05:01:28.560 loader: max number of loaders that ever existed simultaneously: 0
Fri Mar 27 05:01:28.560 memory: number of malloc operations: 0
Fri Mar 27 05:01:28.560 memory: number of free operations: 0
Fri Mar 27 05:01:28.560 memory: number of realloc operations: 0
Fri Mar 27 05:01:28.560 memory: number of malloc operations that failed: 0
Fri Mar 27 05:01:28.560 memory: number of realloc operations that failed: 0
Fri Mar 27 05:01:28.560 memory: number of bytes requested: 0
Fri Mar 27 05:01:28.560 memory: number of bytes used (requested + overhead): 0
Fri Mar 27 05:01:28.560 memory: number of bytes freed: 0
Fri Mar 27 05:01:28.560 memory: largest attempted allocation size: 0
Fri Mar 27 05:01:28.560 memory: size of the last failed allocation attempt: 0
Fri Mar 27 05:01:28.560 memory: estimated maximum memory footprint: 0
Fri Mar 27 05:01:28.560 memory: mallocator version: tokumx-2.0.0-2-0-gc7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 05:01:28.560 memory: mmap threshold: 4194304
Fri Mar 27 05:01:28.560 filesystem: ENOSPC redzone state: 0
Fri Mar 27 05:01:28.560 filesystem: threads currently blocked by full disk: 0
Fri Mar 27 05:01:28.560 filesystem: number of operations rejected by enospc prevention (red zone): 0
Fri Mar 27 05:01:28.560 filesystem: most recent disk full: Thu Jan  1 01:00:00 1970
Fri Mar 27 05:01:28.560 filesystem: number of write operations that returned ENOSPC: 0
Fri Mar 27 05:01:28.560 filesystem: fsync time: 9999231355
Fri Mar 27 05:01:28.560 filesystem: fsync count: 331704
Fri Mar 27 05:01:28.560 filesystem: long fsync time: 0
Fri Mar 27 05:01:28.560 filesystem: long fsync count: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a full fetch: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a partial fetch: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a full eviction: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a partial eviction: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a message injection: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a message application: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a flush: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by a the cleaner thread: 0
Fri Mar 27 05:01:28.560 context: tree traversals blocked by something uninstrumented: 11759
Fri Mar 27 05:01:28.560 context: promotion blocked by a full fetch (should never happen): 0
Fri Mar 27 05:01:28.560 context: promotion blocked by a partial fetch (should never happen): 0
Fri Mar 27 05:01:28.560 context: promotion blocked by a full eviction (should never happen): 0
Fri Mar 27 05:01:28.560 context: promotion blocked by a partial eviction (should never happen): 0
Fri Mar 27 05:01:28.560 context: promotion blocked by a message injection: 35
Fri Mar 27 05:01:28.560 context: promotion blocked by a message application: 0
Fri Mar 27 05:01:28.560 context: promotion blocked by a flush: 0
Fri Mar 27 05:01:28.560 context: promotion blocked by the cleaner thread: 1
Fri Mar 27 05:01:28.560 context: promotion blocked by something uninstrumented: 1
Fri Mar 27 05:01:28.560 context: something uninstrumented blocked by something uninstrumented: 0
Fri Mar 27 05:01:28.560 Memory usage:

Arena 0:
system bytes     =          0
in use bytes     =          0
Total (incl. mmap):
system bytes     =          0
in use bytes     =          0
max mmap regions =          0
max mmap bytes   =          0
Backtrace: (Note: toku_do_assert=0x0x7f7da7f53ce0)
/usr/lib/tokumx/libtokuportability.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f7da7f5319b]
/usr/bin/mongod[0x876f5d]
/usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x37)[0x879167]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x1f1)[0x6d85f1]
/lib/x86_64-linux-gnu/libc.so.6(+0x36c30)[0x7f7da6933c30]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f7da6933bb9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f7da6936fc8]
/usr/lib/tokumx/libtokufractaltree.so(+0x955a2)[0x7f7da81f05a2]
/usr/lib/tokumx/libtokufractaltree.so(+0x958e3)[0x7f7da81f08e3]
/usr/lib/tokumx/libtokufractaltree.so(+0x8feb7)[0x7f7da81eaeb7]
/usr/lib/tokumx/libtokufractaltree.so(+0x90010)[0x7f7da81eb010]
/usr/lib/tokumx/libtokufractaltree.so(+0xa959f)[0x7f7da820459f]
/usr/lib/tokumx/libtokufractaltree.so(+0x3c727)[0x7f7da8197727]
/usr/lib/tokumx/libtokufractaltree.so(+0x3d3d2)[0x7f7da81983d2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f7da846f182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7da69f7efd]
Engine status:
BUILD_ID = 0
time of environment creation: Mon Jan 12 19:57:28 2015
time of engine startup: Thu Mar 26 10:52:21 2015
time now: Fri Mar 27 05:01:28 2015
db opens: 121
db closes: 2
num open dbs now: 119
max open dbs: 119
period, in ms, that recovery log is automatically fsynced: 100
dictionary inserts: 2
dictionary inserts fail: 0
dictionary deletes: 2
dictionary deletes fail: 0
dictionary updates: 2979695
dictionary updates fail: 0
dictionary broadcast updates: 0
dictionary broadcast updates fail: 0
dictionary multi inserts: 3747637
dictionary multi inserts fail: 0
dictionary multi deletes: 2292844
dictionary multi deletes fail: 0
dictionary updates multi: 1916072
dictionary updates multi fail: 0
le: max committed xr: 2
le: max provisional xr: 2
le: expanded: 0
le: max memsize: 3178719
le: size of leafentries before garbage collection (during message application): 26085103964
le: size of leafentries after garbage collection (during message application): 15976507287
le: size of leafentries before garbage collection (outside message application): 0
le: size of leafentries after garbage collection (outside message application): 0
checkpoint: period: 60
checkpoint: footprint: 0
checkpoint: last checkpoint began : Fri Mar 27 05:01:21 2015
checkpoint: last complete checkpoint began : Fri Mar 27 05:01:21 2015
checkpoint: last complete checkpoint ended: Fri Mar 27 05:01:24 2015
checkpoint: time spent during checkpoint (begin and end phases): 3414
checkpoint: time spent during last checkpoint (begin and end phases): 3
checkpoint: last complete checkpoint LSN: 1407606434
checkpoint: checkpoints taken : 1091
checkpoint: checkpoints failed: 0
checkpoint: waiters now: 0
checkpoint: waiters max: 0
checkpoint: non-checkpoint client wait on mo lock: 20
checkpoint: non-checkpoint client wait on cs lock: 0
checkpoint: checkpoint begin time: 1560528
checkpoint: long checkpoint begin time: 0
checkpoint: long checkpoint begin count: 0
cachetable: miss: 6109
cachetable: miss time: 12997796
cachetable: prefetches: 11803
cachetable: size current: 16792927690
cachetable: size limit: 17334624256
cachetable: size writing: 0
cachetable: size nonleaf: 787929143
cachetable: size leaf: 16004997923
cachetable: size rollback: 624
cachetable: size cachepressure: 780495138
cachetable: size currently cloned data for checkpoint: 0
cachetable: evictions: 1089
cachetable: cleaner executions: 35889
cachetable: cleaner period: 2
cachetable: cleaner iterations: 5
cachetable: number of waits on cache pressure: 0
cachetable: time waiting on cache pressure: 0
cachetable: number of long waits on cache pressure: 0
cachetable: long time waiting on cache pressure: 0
locktree: memory size: 0
locktree: memory size limit: 1679775334
locktree: number of times lock escalation ran: 0
locktree: time spent running escalation (seconds): 0.000000
locktree: latest post-escalation memory size: 0
locktree: number of locktrees open now: 121
locktree: number of pending lock requests: 0
locktree: number of locktrees eligible for the STO: 0
locktree: number of times a locktree ended the STO early: 1
locktree: time spent ending the STO early (seconds): 0.000004
locktree: number of wait locks: 0
locktree: time waiting for locks: 0
locktree: number of long wait locks: 0
locktree: long time waiting for locks: 0
locktree: number of lock timeouts: 0
locktree: number of waits on lock escalation: 0
locktree: time waiting on lock escalation: 0
locktree: number of long waits on lock escalation: 0
locktree: long time waiting on lock escalation: 0
ft: dictionary updates: 2975569
ft: dictionary broadcast updates: 0
ft: descriptor set: 2
ft: messages ignored by leaf due to msn: 1246
ft: total search retries due to TRY_AGAIN: 14904
ft: searches requiring more tries than the height of the tree: 0
ft: searches requiring more tries than the height of the tree plus three: 0
ft: leaf nodes flushed to disk (not for checkpoint): 18
ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 7653888
ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 34615817
ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.001496
ft: nonleaf nodes flushed to disk (not for checkpoint): 25
ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 11703296
ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 44346712
ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.002372
ft: leaf nodes flushed to disk (for checkpoint): 29549
ft: leaf nodes flushed to disk (for checkpoint) (bytes): 10838546432
ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 47318770696
ft: leaf nodes flushed to disk (for checkpoint) (seconds): 2.881090
ft: nonleaf nodes flushed to disk (for checkpoint): 38566
ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 13381863424
ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 51681468120
ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 2.894310
ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
ft: nonleaf node partial evictions: 0
ft: nonleaf node partial evictions (bytes): 0
ft: leaf node partial evictions: 633965
ft: leaf node partial evictions (bytes): 51057658086
ft: leaf node full evictions: 901
ft: leaf node full evictions (bytes): 46997044
ft: nonleaf node full evictions: 188
ft: nonleaf node full evictions (bytes): 73773690
ft: leaf nodes created: 1555
ft: nonleaf nodes created: 159
ft: leaf nodes destroyed: 0
ft: nonleaf nodes destroyed: 0
ft: bytes of messages injected at root (all trees): 951158981
ft: bytes of messages flushed from h1 nodes to leaves: 919467528
ft: bytes of messages currently in trees (estimate): 31691453
ft: messages injected at root: 6161984
ft: broadcast messages injected at root: 0
ft: basements decompressed as a target of a query: 1
ft: basements decompressed for prelocked range: 0
ft: basements decompressed for prefetch: 0
ft: basements decompressed for write: 0
ft: buffers decompressed as a target of a query: 0
ft: buffers decompressed for prelocked range: 0
ft: buffers decompressed for prefetch: 0
ft: buffers decompressed for write: 0
ft: pivots fetched for query: 1483
ft: pivots fetched for query (bytes): 48594944
ft: pivots fetched for query (seconds): 0.006302
ft: pivots fetched for prefetch: 11803
ft: pivots fetched for prefetch (bytes): 386760192
ft: pivots fetched for prefetch (seconds): 0.075030
ft: pivots fetched for write: 323
ft: pivots fetched for write (bytes): 5231616
ft: pivots fetched for write (seconds): 0.020985
ft: basements fetched as a target of a query: 1486
ft: basements fetched as a target of a query (bytes): 12849664
ft: basements fetched as a target of a query (seconds): 0.000986
ft: basements fetched for prelocked range: 79143
ft: basements fetched for prelocked range (bytes): 1001997824
ft: basements fetched for prelocked range (seconds): 0.119409
ft: basements fetched for prefetch: 612045
ft: basements fetched for prefetch (bytes): 7546894336
ft: basements fetched for prefetch (seconds): 0.910073
ft: basements fetched for write: 75872
ft: basements fetched for write (bytes): 1524901376
ft: basements fetched for write (seconds): 3.121403
ft: buffers fetched as a target of a query: 0
ft: buffers fetched as a target of a query (bytes): 0
ft: buffers fetched as a target of a query (seconds): 0.000000
ft: buffers fetched for prelocked range: 0
ft: buffers fetched for prelocked range (bytes): 0
ft: buffers fetched for prelocked range (seconds): 0.000000
ft: buffers fetched for prefetch: 0
ft: buffers fetched for prefetch (bytes): 0
ft: buffers fetched for prefetch (seconds): 0.000000
ft: buffers fetched for write: 7953
ft: buffers fetched for write (bytes): 141093888
ft: buffers fetched for write (seconds): 0.023609
ft: leaf compression to memory (seconds): 556.846440
ft: leaf serialization to memory (seconds): 14.886162
ft: leaf decompression to memory (seconds): 77.027153
ft: leaf deserialization to memory (seconds): 8.661888
ft: nonleaf compression to memory (seconds): 682.773049
ft: nonleaf serialization to memory (seconds): 20.804160
ft: nonleaf decompression to memory (seconds): 0.941205
ft: nonleaf deserialization to memory (seconds): 0.362455
ft: promotion: roots split: 6
ft: promotion: leaf roots injected into: 195244
ft: promotion: h1 roots injected into: 111015
ft: promotion: injections at depth 0: 980566
ft: promotion: injections at depth 1: 4724723
ft: promotion: injections at depth 2: 905805
ft: promotion: injections at depth 3: 2093865
ft: promotion: injections lower than depth 3: 1259278
ft: promotion: stopped because of a nonempty buffer: 6008292
ft: promotion: stopped at height 1: 852
ft: promotion: stopped because the child was locked or not at all in memory: 121
ft: promotion: stopped because the child was not fully in memory: 0
ft: promotion: stopped anyway, after locking the child: 614
ft: basement nodes deserialized with fixed-keysize: 744115
ft: basement nodes deserialized with variable-keysize: 24444
ft: promotion: succeeded in using the rightmost leaf shortcut: 2284667
ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 309
ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 723292
ft flusher: total nodes potentially flushed by cleaner thread: 3444
ft flusher: height-one nodes flushed by cleaner thread: 2862
ft flusher: height-greater-than-one nodes flushed by cleaner thread: 582
ft flusher: nodes cleaned which had empty buffers: 0
ft flusher: nodes dirtied by cleaner thread: 6744
ft flusher: max bytes in a buffer flushed by cleaner thread: 3812200
ft flusher: min bytes in a buffer flushed by cleaner thread: 58
ft flusher: total bytes in buffers flushed by cleaner thread: 841217554
ft flusher: max workdone in a buffer flushed by cleaner thread: 1243458
ft flusher: min workdone in a buffer flushed by cleaner thread: 0
ft flusher: total workdone in buffers flushed by cleaner thread: 15229892
ft flusher: times cleaner thread tries to merge a leaf: 24
ft flusher: cleaner thread leaf merges in progress: 0
ft flusher: cleaner thread leaf merges successful: 23
ft flusher: nodes dirtied by cleaner thread leaf merges: 23
ft flusher: total number of flushes done by flusher threads or cleaner threads: 3946
ft flusher: number of in memory flushes: 3946
ft flusher: number of flushes that read something off disk: 0
ft flusher: number of flushes that triggered another flush in child: 67
ft flusher: number of flushes that triggered 1 cascading flush: 64
ft flusher: number of flushes that triggered 2 cascading flushes: 3
ft flusher: number of flushes that triggered 3 cascading flushes: 0
ft flusher: number of flushes that triggered 4 cascading flushes: 0
ft flusher: number of flushes that triggered 5 cascading flushes: 0
ft flusher: number of flushes that triggered over 5 cascading flushes: 0
ft flusher: leaf node splits: 1553
ft flusher: nonleaf node splits: 153
ft flusher: leaf node merges: 23
ft flusher: nonleaf node merges: 0
ft flusher: leaf node balances: 0
hot: operations ever started: 0
hot: operations successfully completed: 0
hot: operations aborted: 0
hot: max number of flushes from root ever required to optimize a tree: 0
txn: begin: 5898911
txn: begin read only: 2177
txn: successful commits: 5899995
txn: aborts: 1093
logger: next LSN: 1407606798
logger: writes: 255197
logger: writes (bytes): 4572456583
logger: writes (uncompressed bytes): 4572456583
logger: writes (seconds): 3.420198
logger: number of long logger write operations: 0
indexer: number of indexers successfully created: 0
indexer: number of calls to toku_indexer_create_indexer() that failed: 0
indexer: number of calls to indexer->build() succeeded: 0
indexer: number of calls to indexer->build() failed: 0
indexer: number of calls to indexer->close() that succeeded: 0
indexer: number of calls to indexer->close() that failed: 0
indexer: number of calls to indexer->abort(): 0
indexer: number of indexers currently in existence: 0
indexer: max number of indexers that ever existed simultaneously: 0
loader: number of loaders successfully created: 0
loader: number of calls to toku_loader_create_loader() that failed: 0
loader: number of calls to loader->put() succeeded: 0
loader: number of calls to loader->put() failed: 0
loader: number of calls to loader->close() that succeeded: 0
loader: number of calls to loader->close() that failed: 0
loader: number of calls to loader->abort(): 0
loader: number of loaders currently in existence: 0
loader: max number of loaders that ever existed simultaneously: 0
memory: number of malloc operations: 0
memory: number of free operations: 0
memory: number of realloc operations: 0
memory: number of malloc operations that failed: 0
memory: number of realloc operations that failed: 0
memory: number of bytes requested: 0
memory: number of bytes used (requested + overhead): 0
memory: number of bytes freed: 0
memory: largest attempted allocation size: 0
memory: size of the last failed allocation attempt: 0
memory: estimated maximum memory footprint: 0
memory: mallocator version: tokumx-2.0.0-2-0-gc7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
memory: mmap threshold: 4194304
filesystem: ENOSPC redzone state: 0
filesystem: threads currently blocked by full disk: 0
filesystem: number of operations rejected by enospc prevention (red zone): 0
filesystem: most recent disk full: Thu Jan  1 01:00:00 1970
filesystem: number of write operations that returned ENOSPC: 0
filesystem: fsync time: 9999231355
filesystem: fsync count: 331704
filesystem: long fsync time: 0
filesystem: long fsync count: 0
context: tree traversals blocked by a full fetch: 0
context: tree traversals blocked by a partial fetch: 0
context: tree traversals blocked by a full eviction: 0
context: tree traversals blocked by a partial eviction: 0
context: tree traversals blocked by a message injection: 0
context: tree traversals blocked by a message application: 0
context: tree traversals blocked by a flush: 0
context: tree traversals blocked by a the cleaner thread: 0
context: tree traversals blocked by something uninstrumented: 11759
context: promotion blocked by a full fetch (should never happen): 0
context: promotion blocked by a partial fetch (should never happen): 0
context: promotion blocked by a full eviction (should never happen): 0
context: promotion blocked by a partial eviction (should never happen): 0
context: promotion blocked by a message injection: 35
context: promotion blocked by a message application: 0
context: promotion blocked by a flush: 0
context: promotion blocked by the cleaner thread: 1
context: promotion blocked by something uninstrumented: 1
context: something uninstrumented blocked by something uninstrumented: 0

Memory usage:
Arena 0:
system bytes     =          0
in use bytes     =          0
Total (incl. mmap):
system bytes     =          0
in use bytes     =          0
max mmap regions =          0
max mmap bytes   =          0
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.561 Crash reason:
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 Got signal: 6 (Aborted).
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.561 Process info:
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 OS:   Linux Ubuntu 14.04 x86_64
Fri Mar 27 05:01:28.561 NCPU: 8
Fri Mar 27 05:01:28.561 VIRT: 27806 MB
Fri Mar 27 05:01:28.561 RES:  27806 MB
Fri Mar 27 05:01:28.561 PHYS: 32039 MB
Fri Mar 27 05:01:28.561 RLIMIT_CORE: 0 (soft), unlimited (hard)
Fri Mar 27 05:01:28.561 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Fri Mar 27 05:01:28.561 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Fri Mar 27 05:01:28.561 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Fri Mar 27 05:01:28.561 RLIMIT_NOFILE: 1024 (soft), 4096 (hard)
Fri Mar 27 05:01:28.561 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Fri Mar 27 05:01:28.561 RLIMIT_AS: unlimited (soft), unlimited (hard)
Fri Mar 27 05:01:28.561 _SC_OPEN_MAX: 1024
Fri Mar 27 05:01:28.561 _SC_PAGESIZE: 4096
Fri Mar 27 05:01:28.561 _SC_PHYS_PAGES: 8202028
Fri Mar 27 05:01:28.561 _SC_AVPHYS_PAGES: 306389
Fri Mar 27 05:01:28.561 _SC_NPROCESSORS_CONF: 8
Fri Mar 27 05:01:28.561 _SC_NPROCESSORS_ONLN: 8
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.561 Parsed server options:
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 config: "/etc/tokumx.conf"
Fri Mar 27 05:01:28.561 dbpath: "/var/lib/tokumx"
Fri Mar 27 05:01:28.561 expireOplogDays: 14
Fri Mar 27 05:01:28.561 logappend: "true"
Fri Mar 27 05:01:28.561 logpath: "/var/log/tokumx/tokumx.log"
Fri Mar 27 05:01:28.561 pluginsDir: "/usr/lib/tokumx/plugins"
Fri Mar 27 05:01:28.561 port: 23456
Fri Mar 27 05:01:28.561 replSet: "tktoku"
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.561 Filesystem information:
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 Information for dbpath "/var/lib/tokumx":
Fri Mar 27 05:01:28.561  
Fri Mar 27 05:01:28.561 type magic: 0x0000EF53
Fri Mar 27 05:01:28.561 type: ext2/3/4
Fri Mar 27 05:01:28.561 bsize: 4096
Fri Mar 27 05:01:28.561 blocks: 53364398
Fri Mar 27 05:01:28.562 bfree: 32881342
Fri Mar 27 05:01:28.562 bavail: 30164818
Fri Mar 27 05:01:28.562  
Fri Mar 27 05:01:28.562 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.562 Current operations in progress:
Fri Mar 27 05:01:28.562  
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "local.oplog.rs", query: {}, desc: "applier", threadId: "0x7f79eb3ff700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 128446748, w: 0 }, timeAcquiringMicros: { r: 5081236, w: 0 } } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "local.oplog.rs", query: {}, desc: "oplogPartitionThread", threadId: "0x7f794d3fc700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 26782, w: 271184 }, timeAcquiringMicros: { r: 196322, w: 2 } } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "keepOplogAlive", threadId: "0x7f794dbfd700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "rsGhostSync", threadId: "0x7f794cbfb700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 105469, active: false, op: "query", ns: "", query: { serverStatus: 1 }, client: "127.0.0.1:58669", desc: "conn5", threadId: "0x7f7da55f2700", connectionId: 5, rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 105468, active: false, op: "query", ns: "", query: { replSetHeartbeat: "tktoku", v: 3, pv: 65, checkEmpty: false, from: "toku-atk2:23456" }, client: "10.1.0.42:39844", desc: "conn7522", threadId: "0x7f7d753f4700", connectionId: 7522, rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "rsMgr", threadId: "0x7f7d75bf5700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "clientcursormon", threadId: "0x7f7d78bfb700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 105466, active: false, op: "query", ns: "local.system.indexes", query: { expireAfterSeconds: { $exists: true } }, client: "0.0.0.0:0", desc: "TTLMonitor", threadId: "0x7f7d77bf9700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 22, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "snapshotthread", threadId: "0x7f7d793fc700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "websvr", threadId: "0x7f7d76bf7700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "local.oplog.refs", query: {}, desc: "updateReplInfo", threadId: "0x7f794e3fe700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 1133188, w: 0 }, timeAcquiringMicros: { r: 312065, w: 0 } } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "insert", ns: "local.startup_log", insert: {}, client: "0.0.0.0:0", desc: "initandlisten", threadId: "0x7f7da9335880", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 86, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 05:01:28.562 { opid: 0, active: false, op: "none", ns: "local.oplog.rs", query: {}, desc: "rsBackgroundSync", threadId: "0x7f794ebff700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 35924193, w: 0 }, timeAcquiringMicros: { r: 4010078, w: 0 } } }
Fri Mar 27 05:01:28.562  
Fri Mar 27 05:01:28.562 --------------------------------------------------------------------------------
Fri Mar 27 05:01:28.562 OpDebug info:
Fri Mar 27 05:01:28.562  
Fri Mar 27 05:01:28.562 none  keyUpdates:0 locks(micros) r:128446748 0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0 locks(micros) r:26782 w:271184 0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0 locks(micros) r:22 0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0  0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0 locks(micros) r:1133188 0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0 locks(micros) r:86 0ms
Fri Mar 27 05:01:28.562 none  keyUpdates:0 locks(micros) r:35924193 0ms
Fri Mar 27 05:01:28.562

last crash, after I saw the notifications, found it dead and attempted to start it:

Fri Mar 27 06:53:33.592 [initandlisten] TokuMX starting : pid=20158 port=23456 dbpath=/var/lib/tokumx 64-bit host=toku-atk1
Fri Mar 27 06:53:33.592 [initandlisten] TokuMX mongod server v2.0.0-mongodb-2.4.10, using TokuKV rev 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 06:53:33.592 [initandlisten] git version: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 06:53:33.592 [initandlisten] build info: Linux 3eb534264b3c 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_54
Fri Mar 27 06:53:33.592 [initandlisten] options: { config: "/etc/tokumx.conf", dbpath: "/var/lib/tokumx", expireOplogDays: 14, logappend: "true", logpath: "/var/log/tokumx/tokumx.log", plugin
sDir: "/usr/lib/tokumx/plugins", port: 23456, replSet: "tktoku" }
Fri Mar 27 06:53:33.640 [initandlisten] [tokumx] startup
Fri Mar 27 06:53:33 2015 TokuFT recovery starting in env /var/lib/tokumx
Fri Mar 27 06:53:33 2015 TokuFT recovery scanning backward from 1408857577
Fri Mar 27 06:53:33 2015 TokuFT recovery bw_end_checkpoint at 1408856850 timestamp 1427434116363329 xid 1408856824 (bw_newer)
Fri Mar 27 06:53:33 2015 TokuFT recovery bw_begin_checkpoint at 1408856824 timestamp 1427434116001057 (bw_between)
Fri Mar 27 06:53:33 2015 TokuFT recovery turning around at begin checkpoint 1408856824 time 362272
Fri Mar 27 06:53:33 2015 TokuFT recovery starts scanning forward to 1408857577 from 1408856824 left 753 (fw_between)
Fri Mar 27 06:53:33 2015 TokuFT recovery has 1 live transaction
Fri Mar 27 06:53:33 2015 TokuFT recovery closing 29 dictionaries
Checksum failure while reading node partition in file somecollection.tokumx.
Fri Mar 27 06:53:33.787  
Fri Mar 27 06:53:33.787 ================================================================================
Fri Mar 27 06:53:33.787  Fatal error detected
Fri Mar 27 06:53:33.787 ================================================================================
Fri Mar 27 06:53:33.787  
Fri Mar 27 06:53:33.787 About to gather debugging information, please include all of the following along
Fri Mar 27 06:53:33.787 with logs from other servers in the cluster in a bug report.
Fri Mar 27 06:53:33.787  
Fri Mar 27 06:53:33.787 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.787 Version info:
Fri Mar 27 06:53:33.787  
Fri Mar 27 06:53:33.787 tokumxVersion: 2.0.0
Fri Mar 27 06:53:33.787 gitVersion: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 06:53:33.787 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 06:53:33.787 sysInfo: Linux 3eb534264b3c 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_54
Fri Mar 27 06:53:33.787 loaderFlags:        
Fri Mar 27 06:53:33.787 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local
-typedefs -fno-builtin-memcmp -O3
Fri Mar 27 06:53:33.787 debug: false
Fri Mar 27 06:53:33.787  
Fri Mar 27 06:53:33.787 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.787 Simple stacktrace:
Fri Mar 27 06:53:33.787  
0xb0d7f6 0x876e44 0x879162 0x6d85f1 0x7facea5fbc30 0x7facea5fbbb9 0x7facea5fefc8 0x7facebeb85a2 0x7facebeb88e3 0x7facebeb2eb7 0x7facebeb3010 0x7facebeb3010 0x7facebeb417e 0x7facebe426e4 0x7facec137182 0x7facea6bfefd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xb0d7f6]
 /usr/bin/mongod() [0x876e44]
 /usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x32) [0x879162]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x1f1) [0x6d85f1]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7facea5fbc30]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7facea5fbbb9]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7facea5fefc8]
 /usr/lib/tokumx/libtokufractaltree.so(+0x955a2) [0x7facebeb85a2]
 /usr/lib/tokumx/libtokufractaltree.so(+0x958e3) [0x7facebeb88e3]
 /usr/lib/tokumx/libtokufractaltree.so(+0x8feb7) [0x7facebeb2eb7]
 /usr/lib/tokumx/libtokufractaltree.so(+0x90010) [0x7facebeb3010]
 /usr/lib/tokumx/libtokufractaltree.so(+0x90010) [0x7facebeb3010]
 /usr/lib/tokumx/libtokufractaltree.so(+0x9117e) [0x7facebeb417e]
 /usr/lib/tokumx/libtokufractaltree.so(+0x1f6e4) [0x7facebe426e4]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7facec137182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7facea6bfefd]
Fri Mar 27 06:53:33.788  
Fri Mar 27 06:53:33.788 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.788 TokuKV engine backtrace:
Fri Mar 27 06:53:33.788  
Fri Mar 27 06:53:33.788 Backtrace: (Note: toku_do_assert=0x0x7facebc1bce0)

Fri Mar 27 06:53:33.788 /usr/lib/tokumx/libtokuportability.so(_Z27db_env_do_backtrace_errfuncPFvPKviPKczES0_+0x1f) [0x7facebc1d5bf]
Fri Mar 27 06:53:33.788 /usr/lib/tokumx/libtokufractaltree.so(+0x149cd) [0x7facebe379cd]
Fri Mar 27 06:53:33.788 /usr/bin/mongod() [0x876f5d]
Fri Mar 27 06:53:33.788 /usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x37) [0x879167]
Fri Mar 27 06:53:33.788 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x1f1) [0x6d85f1]
Fri Mar 27 06:53:33.788 /lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7facea5fbc30]
Fri Mar 27 06:53:33.788 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7facea5fbbb9]
Fri Mar 27 06:53:33.789 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7facea5fefc8]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x955a2) [0x7facebeb85a2]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x958e3) [0x7facebeb88e3]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x8feb7) [0x7facebeb2eb7]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x90010) [0x7facebeb3010]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x90010) [0x7facebeb3010]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x9117e) [0x7facebeb417e]
Fri Mar 27 06:53:33.789 /usr/lib/tokumx/libtokufractaltree.so(+0x1f6e4) [0x7facebe426e4]
Fri Mar 27 06:53:33.789 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7facec137182]
Fri Mar 27 06:53:33.789 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7facea6bfefd]
Fri Mar 27 06:53:33.789 Engine status function not available

Fri Mar 27 06:53:33.789 Memory usage:

Arena 0:
system bytes     =          0
in use bytes     =          0
Total (incl. mmap):
system bytes     =          0
in use bytes     =          0
Total (incl. mmap):
system bytes     =          0
in use bytes     =          0
max mmap regions =          0
max mmap bytes   =          0
Backtrace: (Note: toku_do_assert=0x0x7facebc1bce0)
/usr/lib/tokumx/libtokuportability.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7facebc1b19b]
/usr/bin/mongod[0x876f5d]
/usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x37)[0x879167]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x1f1)[0x6d85f1]
/lib/x86_64-linux-gnu/libc.so.6(+0x36c30)[0x7facea5fbc30]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7facea5fbbb9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7facea5fefc8]
/usr/lib/tokumx/libtokufractaltree.so(+0x955a2)[0x7facebeb85a2]
/usr/lib/tokumx/libtokufractaltree.so(+0x958e3)[0x7facebeb88e3]
/usr/lib/tokumx/libtokufractaltree.so(+0x8feb7)[0x7facebeb2eb7]
/usr/lib/tokumx/libtokufractaltree.so(+0x90010)[0x7facebeb3010]
/usr/lib/tokumx/libtokufractaltree.so(+0x90010)[0x7facebeb3010]
/usr/lib/tokumx/libtokufractaltree.so(+0x9117e)[0x7facebeb417e]
/usr/lib/tokumx/libtokufractaltree.so(+0x1f6e4)[0x7facebe426e4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7facec137182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7facea6bfefd]
Engine status function not available
Memory usage:
Arena 0:
system bytes     =          0
in use bytes     =          0
Total (incl. mmap):
system bytes     =          0
in use bytes     =          0
max mmap regions =          0
max mmap bytes   =          0
Fri Mar 27 06:53:33.789  
Fri Mar 27 06:53:33.789 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.789 Crash reason:
Fri Mar 27 06:53:33.789  
Fri Mar 27 06:53:33.789 Got signal: 6 (Aborted).
Fri Mar 27 06:53:33.789  
Fri Mar 27 06:53:33.789 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.789 Process info:
Fri Mar 27 06:53:33.789  
Fri Mar 27 06:53:33.789 OS:   Linux Ubuntu 14.04 x86_64
Fri Mar 27 06:53:33.789 NCPU: 8
Fri Mar 27 06:53:33.790 VIRT: 481 MB
Fri Mar 27 06:53:33.790 RES:  481 MB
Fri Mar 27 06:53:33.790 PHYS: 32039 MB
Fri Mar 27 06:53:33.790 RLIMIT_CORE: 0 (soft), unlimited (hard)
Fri Mar 27 06:53:33.790 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Fri Mar 27 06:53:33.790 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Fri Mar 27 06:53:33.790 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Fri Mar 27 06:53:33.790 RLIMIT_NOFILE: 1024 (soft), 4096 (hard)
Fri Mar 27 06:53:33.790 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Fri Mar 27 06:53:33.790 RLIMIT_AS: unlimited (soft), unlimited (hard)
Fri Mar 27 06:53:33.790 _SC_OPEN_MAX: 1024
Fri Mar 27 06:53:33.790 _SC_PAGESIZE: 4096
Fri Mar 27 06:53:33.790 _SC_PHYS_PAGES: 8202028
Fri Mar 27 06:53:33.790 _SC_AVPHYS_PAGES: 4539629
Fri Mar 27 06:53:33.790 _SC_NPROCESSORS_CONF: 8
Fri Mar 27 06:53:33.790 _SC_NPROCESSORS_ONLN: 8
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.790 Parsed server options:
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 config: "/etc/tokumx.conf"
Fri Mar 27 06:53:33.790 dbpath: "/var/lib/tokumx"
Fri Mar 27 06:53:33.790 expireOplogDays: 14
Fri Mar 27 06:53:33.790 logappend: "true"
Fri Mar 27 06:53:33.790 logpath: "/var/log/tokumx/tokumx.log"
Fri Mar 27 06:53:33.790 pluginsDir: "/usr/lib/tokumx/plugins"
Fri Mar 27 06:53:33.790 port: 23456
Fri Mar 27 06:53:33.790 replSet: "tktoku"
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.790 Filesystem information:
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 Information for dbpath "/var/lib/tokumx":
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 type magic: 0x0000EF53
Fri Mar 27 06:53:33.790 type: ext2/3/4
Fri Mar 27 06:53:33.790 bsize: 4096
Fri Mar 27 06:53:33.790 blocks: 53364398
Fri Mar 27 06:53:33.790 bfree: 32886885
Fri Mar 27 06:53:33.790 bavail: 30170361
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.790 Current operations in progress:
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "initandlisten", threadId: "0x7facecffd880", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 --------------------------------------------------------------------------------
Fri Mar 27 06:53:33.790 OpDebug info:
Fri Mar 27 06:53:33.790  
Fri Mar 27 06:53:33.790 none  keyUpdates:0  0ms
Fri Mar 27 06:53:33.790
adeelahmadch commented 9 years ago

its very sad to see tokumx just crashed, it should handle the error instead of crashing. i ran into same issue and in my case i found out the reason i.e. my code tries to build an index on very large collection and after 98% it just crashed with same error exception: i tried to increase ulimit , upgrade hardware. but its looks like its the issue with tokumx even in my updated version.

Fri Mar 27 05:01:28.561 Crash reason: Fri Mar 27 05:01:28.561
Fri Mar 27 05:01:28.561 Got signal: 6 (Aborted)..

i am using 2.0.1. you can see my issue logged here: https://github.com/Tokutek/mongo/issues/1228

m0n5t3r commented 9 years ago

we bit the bullet and moved back to mongodb (and upgraded to 3.0 with WiredTiger, which is giving us similar performance and only slightly larger disk usage), 6h of downtime and all... this thing seems dead (they were acquired by Percona some time ago AFAIR, I don't see them pursuing the mongodb thing further)

On 06/04/2015 04:14 PM, Adeel Ahmad wrote:

its very sad to see tokumx is just crashing, it should handle the error instead of crashing. i ran into same issue. in my case i found out the reason, my code tries to build an index on very large collection and after 98% it just crashed with same error exception, Fri Mar 27 05:01:28.561 Crash reason: Fri Mar 27 05:01:28.561

Fri Mar 27 05:01:28.561 Got signal: 6 (Aborted)..

i am using 2.0.1.

you can see my issue logged here: #1228 https://github.com/Tokutek/mongo/issues/1228

— Reply to this email directly or view it on GitHub https://github.com/Tokutek/mongo/issues/1219#issuecomment-108893952.

adeelahmadch commented 9 years ago

Thank you very much, i am also trying to test my DB with Mongo 3.0.

i think now is the time to move again from TokuMX to MongoDB :-)

aerrami commented 9 years ago

Hello Adeel, Sabin: you are right Percona has acquired Tokutek https://www.percona.com/percona-acquires-tokutek but your statement "I don't see them pursuing the mongodb thing further" is NOT TRUEHere is an official statement from the announcement that we made. WHAT HAPPENS TO TOKUMX?Percona will now offer TokuMX as a fully ACID-compliant drop in alternate replacement to MongoDB®, fully backed by Percona’s renowned support and service offerings. Percona’s TokuMX will provide MongoDB® users with a well known and trusted alternative to existing Support and Consulting services providers. Adeel: It looks like a bug I'm working on a reproducer, it will be nice if you can provide one or your datafrom the data you have provided it looks like your are running out of memory

cacheSize: 5368709120 which smaller that cachetable: size current: 5486165566

your virtual memory is set to ~7G  so as workaround  you may want to lower cacheSize to like  3G  or increase the size of the virtual memory Regards,Abdelhak ErramiSr Support Engineer | Tokutek a Percona companyabdelhak@tokutek.comabdelhak.errami@percona.com

 On Thursday, June 4, 2015 9:26 AM, Adeel Ahmad <notifications@github.com> wrote:

Thank you very much, i am also trying to test my DB with Mongo 3.0. i think now is the time to move again from TokuMX to MongoDB :-)— Reply to this email directly or view it on GitHub.

adeelahmadch commented 9 years ago

Thank you for replying, i spend last 18 hour and i can reproduce it anywhere and i know its the bug and tokumx should handle that instead of crashing. i can not provide you the data farm but i can help you in providing stats.

i upgraded my hardware , even changed my OS to CentoS 6.6. below are my test result. i tried with 30 GB Physical Memory and with 150GB SWAP and it failed, so i guess its not the virtual memory issue.

1st Test Run for index building

Hard Ware: Amazon Linux , R3.xlarge(4vCPU with 32GB RAM + 150GB SWAP) Disk Space more than 200GB free Space. TokuMX: 2.0.1 Collection catalog_item Size: 108GB Index: db.catalog_item.createIndex( { newItemExtQuoteId : 1 } , { sparse: true }) Ulimit: 64000 Result: TokuMX Crashed.

2nd Test Run for index building

CentOS 6.6, R3.xlarge(4vCPU with 32GB RAM + 150GB SWAP) Disk: More than 500 GB Free Space. TokuMX: 2.0.1 Ulimit: 64000 Collection catalog_item Size: 108GB Index: db.catalog_item.createIndex( { newItemExtQuoteId : 1 } , { sparse: true }) Result: TokuMX Crashed.

adeelahmadch commented 9 years ago

@ aerrami:

i tested it further its a bug in tokumx. now i use Server with 120GB RAM and 500GB HDD. i didn't specify cache size which means that line is comment in tokumx.conf. i tried to build the index and it crashed.

what i found is:

  1. if i try to build index in foreground on large collection , tokumx crashes. db.catalog_item.createIndex( { newItemExtQuoteId : 1 } , { sparse: true })
  2. if we build index in background on large collection it worked fine and consume less resources. in my case i was able to that on server with 15GB RAM and 5GB cache size assigned to tokumx. db.catalog_item.createIndex( { newItemExtQuoteId : 1 } , { sparse: true, background: true })

i am expecting more from tokumx, crashing is not the option, it should throw some kind of error instead of crashing.