We have a replica set with 2 replicas + arbiter. This morning, one of the replicas crashed (and won't start again, will file a separate issue), and the other one works mostly fine, but I can't rebuild the replica set because listDatabases makes the thing crash with [1];
<rant>even more awesomer, there is absolutely no repair option I can find, because this code can't ever be wrong and filesystems can't ever get corrupted...</rant>
[1]
Fri Mar 27 11:06:06.395 nssize is a deprecated parameter
Fri Mar 27 11:06:06.397 [initandlisten] TokuMX starting : pid=29832 port=13456 dbpath=atk2/tokumx-atk2.bak/ 64-bit host=toku-bk1
Fri Mar 27 11:06:06.397 [initandlisten] TokuMX mongod server v2.0.0-mongodb-2.4.10, using TokuKV rev 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 11:06:06.397 [initandlisten] git version: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:06.397 [initandlisten] build info: Linux c1ad9bbe58cd 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Fri Mar 27 11:06:06.397 [initandlisten] options: { dbpath: "atk2/tokumx-atk2.bak/", port: 13456 }
Fri Mar 27 11:06:06.417 [initandlisten] [tokumx] startup
Fri Mar 27 11:06:06.968 [initandlisten]
Fri Mar 27 11:06:06.968 [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset
Fri Mar 27 11:06:06.968 [initandlisten] ** Restart with --replSet unless you are doing maintenance and no other clients are connected.
Fri Mar 27 11:06:06.968 [initandlisten] ** The TTL collection monitor will not start because of this.
Fri Mar 27 11:06:06.968 [initandlisten] ** For more info see http://dochub.mongodb.org/core/ttlcollections
Fri Mar 27 11:06:06.968 [initandlisten]
Fri Mar 27 11:06:06.968 [initandlisten] waiting for connections on port 13456
Fri Mar 27 11:06:06.969 [websvr] admin web console waiting for connections on port 14456
Fri Mar 27 11:06:10.712 [initandlisten] connection accepted from 127.0.0.1:57521 #1 (1 connection now open)
Fri Mar 27 11:06:12.496 [conn1] lock status: R recursive:2 otherCount:0
Fri Mar 27 11:06:12.496 [conn1] Assertion: 16105:expected to be write locked for maindb.system.users
Fri Mar 27 11:06:12.502 [conn1] maindb.system.namespaces Assertion failure !Lock::isReadLocked() /data/release_build-linux-c-opt/build/src/mongo/db/txn_complete_hooks.cpp 67
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 ================================================================================
Fri Mar 27 11:06:12.508 Fatal error detected
Fri Mar 27 11:06:12.508 ================================================================================
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 About to gather debugging information, please include all of the following along
Fri Mar 27 11:06:12.508 with logs from other servers in the cluster in a bug report.
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.508 Version info:
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 tokumxVersion: 2.0.0
Fri Mar 27 11:06:12.508 gitVersion: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.508 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 11:06:12.508 sysInfo: Linux c1ad9bbe58cd 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Fri Mar 27 11:06:12.508 loaderFlags:
Fri Mar 27 11:06:12.508 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local-typedefs -fno-builtin-memcmp -O3
Fri Mar 27 11:06:12.508 debug: false
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.508 Simple stacktrace:
Fri Mar 27 11:06:12.508
0xc4efe3 0x953177 0x955fbb 0x776900 0x7f05ac0f9836 0x7f05ac0f88f9 0x7f05ac0f94aa 0x7f05abb8eff3 0x7f05abb8f517 0x86b3a6 0x86b3d5 0x967a31 0x980974 0x980b1d 0xa090bc 0xa2dccf 0xa1b829 0xa1efc7 0xa1f8ca 0x962c35
/opt/tokumx/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4efe3]
/opt/tokumx/bin/mongod() [0x953177]
/opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
/opt/tokumx/bin/mongod(_ZN5mongo11myterminateEv+0x20) [0x776900]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
/lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
/opt/tokumx/bin/mongod() [0x86b3a6]
/opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
/opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
/opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
/opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
/opt/tokumx/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x39) [0xa1b829]
/opt/tokumx/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcd7) [0xa1efc7]
/opt/tokumx/bin/mongod(_ZN5mongo12_runCommandsEPKcRKNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3aa) [0xa1f8ca]
/opt/tokumx/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x962c35]
Fri Mar 27 11:06:12.513
Fri Mar 27 11:06:12.513 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.513 TokuKV engine backtrace:
Fri Mar 27 11:06:12.513
Fri Mar 27 11:06:12.513 Backtrace: (Note: toku_do_assert=0x0x7f05acbe68b0)
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/../lib64/libtokuportability.so(_Z27db_env_do_backtrace_errfuncPFvPKviPKczES0_+0x1f) [0x7f05acbe7c9f]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/../lib64/libtokufractaltree.so(+0x1437d) [0x7f05ace0137d]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod() [0x953248]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo11myterminateEv+0x20) [0x776900]
Fri Mar 27 11:06:12.518 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
Fri Mar 27 11:06:12.518 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
Fri Mar 27 11:06:12.518 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod() [0x86b3a6]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x39) [0xa1b829]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcd7) [0xa1efc7]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo12_runCommandsEPKcRKNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3aa) [0xa1f8ca]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x962c35]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x6e3) [0x9663b3]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6cb) [0x993a5b]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x8b) [0x78cd3b]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x3dc) [0x89649c]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f05ac7c8182]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f05ab8b400d]
Fri Mar 27 11:06:12.518 BUILD_ID = 107540753
Fri Mar 27 11:06:12.518 time of environment creation: Tue Jan 13 07:26:51 2015
Fri Mar 27 11:06:12.518 time of engine startup: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 time now: Fri Mar 27 11:06:12 2015
Fri Mar 27 11:06:12.518 db opens: 440
Fri Mar 27 11:06:12.518 db closes: 3
Fri Mar 27 11:06:12.518 num open dbs now: 437
Fri Mar 27 11:06:12.518 max open dbs: 440
Fri Mar 27 11:06:12.518 period, in ms, that recovery log is automatically fsynced: 100
Fri Mar 27 11:06:12.518 dictionary inserts: 0
Fri Mar 27 11:06:12.518 dictionary inserts fail: 0
Fri Mar 27 11:06:12.518 dictionary deletes: 1
Fri Mar 27 11:06:12.518 dictionary deletes fail: 0
Fri Mar 27 11:06:12.518 dictionary updates: 0
Fri Mar 27 11:06:12.518 dictionary updates fail: 0
Fri Mar 27 11:06:12.518 dictionary broadcast updates: 0
Fri Mar 27 11:06:12.518 dictionary broadcast updates fail: 0
Fri Mar 27 11:06:12.518 dictionary multi inserts: 1
Fri Mar 27 11:06:12.518 dictionary multi inserts fail: 0
Fri Mar 27 11:06:12.518 dictionary multi deletes: 2
Fri Mar 27 11:06:12.518 dictionary multi deletes fail: 0
Fri Mar 27 11:06:12.518 dictionary updates multi: 0
Fri Mar 27 11:06:12.518 dictionary updates multi fail: 0
Fri Mar 27 11:06:12.518 le: max committed xr: 1
Fri Mar 27 11:06:12.518 le: max provisional xr: 2
Fri Mar 27 11:06:12.518 le: expanded: 0
Fri Mar 27 11:06:12.518 le: max memsize: 883
Fri Mar 27 11:06:12.518 le: size of leafentries before garbage collection (during message application): 0
Fri Mar 27 11:06:12.518 le: size of leafentries after garbage collection (during message application): 0
Fri Mar 27 11:06:12.518 le: size of leafentries before garbage collection (outside message application): 0
Fri Mar 27 11:06:12.518 le: size of leafentries after garbage collection (outside message application): 0
Fri Mar 27 11:06:12.518 checkpoint: period: 60
Fri Mar 27 11:06:12.518 checkpoint: footprint: 0
Fri Mar 27 11:06:12.518 checkpoint: last checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 checkpoint: last complete checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 checkpoint: last complete checkpoint ended: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 checkpoint: time spent during checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.518 checkpoint: time spent during last checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.518 checkpoint: last complete checkpoint LSN: 2565863732
Fri Mar 27 11:06:12.518 checkpoint: checkpoints taken : 2
Fri Mar 27 11:06:12.518 checkpoint: checkpoints failed: 0
Fri Mar 27 11:06:12.518 checkpoint: waiters now: 0
Fri Mar 27 11:06:12.518 checkpoint: waiters max: 0
Fri Mar 27 11:06:12.518 checkpoint: non-checkpoint client wait on mo lock: 0
Fri Mar 27 11:06:12.518 checkpoint: non-checkpoint client wait on cs lock: 0
Fri Mar 27 11:06:12.518 checkpoint: checkpoint begin time: 15
Fri Mar 27 11:06:12.518 checkpoint: long checkpoint begin time: 0
Fri Mar 27 11:06:12.518 checkpoint: long checkpoint begin count: 0
Fri Mar 27 11:06:12.518 cachetable: miss: 12
Fri Mar 27 11:06:12.518 cachetable: miss time: 644
Fri Mar 27 11:06:12.518 cachetable: prefetches: 0
Fri Mar 27 11:06:12.518 cachetable: size current: 406407
Fri Mar 27 11:06:12.518 cachetable: size limit: 17334624256
Fri Mar 27 11:06:12.518 cachetable: size writing: 0
Fri Mar 27 11:06:12.518 cachetable: size nonleaf: 0
Fri Mar 27 11:06:12.518 cachetable: size leaf: 403919
Fri Mar 27 11:06:12.518 cachetable: size rollback: 2488
Fri Mar 27 11:06:12.518 cachetable: size cachepressure: 0
Fri Mar 27 11:06:12.518 cachetable: size currently cloned data for checkpoint: 0
Fri Mar 27 11:06:12.518 cachetable: evictions: 0
Fri Mar 27 11:06:12.518 cachetable: cleaner executions: 3
Fri Mar 27 11:06:12.518 cachetable: cleaner period: 2
Fri Mar 27 11:06:12.518 cachetable: cleaner iterations: 5
Fri Mar 27 11:06:12.518 cachetable: number of waits on cache pressure: 0
Fri Mar 27 11:06:12.518 cachetable: time waiting on cache pressure: 0
Fri Mar 27 11:06:12.518 cachetable: number of long waits on cache pressure: 0
Fri Mar 27 11:06:12.518 cachetable: long time waiting on cache pressure: 0
Fri Mar 27 11:06:12.518 locktree: memory size: 576
Fri Mar 27 11:06:12.518 locktree: memory size limit: 1679775334
Fri Mar 27 11:06:12.518 locktree: number of times lock escalation ran: 0
Fri Mar 27 11:06:12.518 locktree: time spent running escalation (seconds): 0.000000
Fri Mar 27 11:06:12.518 locktree: latest post-escalation memory size: 0
Fri Mar 27 11:06:12.518 locktree: number of locktrees open now: 440
Fri Mar 27 11:06:12.518 locktree: number of pending lock requests: 0
Fri Mar 27 11:06:12.518 locktree: number of locktrees eligible for the STO: 5
Fri Mar 27 11:06:12.518 locktree: number of times a locktree ended the STO early: 0
Fri Mar 27 11:06:12.518 locktree: time spent ending the STO early (seconds): 0.000000
Fri Mar 27 11:06:12.518 locktree: number of wait locks: 0
Fri Mar 27 11:06:12.518 locktree: time waiting for locks: 0
Fri Mar 27 11:06:12.518 locktree: number of long wait locks: 0
Fri Mar 27 11:06:12.518 locktree: long time waiting for locks: 0
Fri Mar 27 11:06:12.518 locktree: number of lock timeouts: 0
Fri Mar 27 11:06:12.518 locktree: number of waits on lock escalation: 0
Fri Mar 27 11:06:12.518 locktree: time waiting on lock escalation: 0
Fri Mar 27 11:06:12.518 locktree: number of long waits on lock escalation: 0
Fri Mar 27 11:06:12.518 locktree: long time waiting on lock escalation: 0
Fri Mar 27 11:06:12.518 ft: dictionary updates: 0
Fri Mar 27 11:06:12.518 ft: dictionary broadcast updates: 0
Fri Mar 27 11:06:12.518 ft: descriptor set: 0
Fri Mar 27 11:06:12.518 ft: messages ignored by leaf due to msn: 0
Fri Mar 27 11:06:12.518 ft: total search retries due to TRY_AGAIN: 0
Fri Mar 27 11:06:12.518 ft: searches requiring more tries than the height of the tree: 0
Fri Mar 27 11:06:12.518 ft: searches requiring more tries than the height of the tree plus three: 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint): 4
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 35840
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 187286
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.000023
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.518 ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.518 ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.518 ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.519 ft: nonleaf node partial evictions: 0
Fri Mar 27 11:06:12.519 ft: nonleaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: leaf node partial evictions: 0
Fri Mar 27 11:06:12.519 ft: leaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: leaf node full evictions: 0
Fri Mar 27 11:06:12.519 ft: leaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: nonleaf node full evictions: 0
Fri Mar 27 11:06:12.519 ft: nonleaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: leaf nodes created: 0
Fri Mar 27 11:06:12.519 ft: nonleaf nodes created: 0
Fri Mar 27 11:06:12.519 ft: leaf nodes destroyed: 0
Fri Mar 27 11:06:12.519 ft: nonleaf nodes destroyed: 0
Fri Mar 27 11:06:12.519 ft: bytes of messages injected at root (all trees): 0
Fri Mar 27 11:06:12.519 ft: bytes of messages flushed from h1 nodes to leaves: 0
Fri Mar 27 11:06:12.519 ft: bytes of messages currently in trees (estimate): 0
Fri Mar 27 11:06:12.519 ft: messages injected at root: 0
Fri Mar 27 11:06:12.519 ft: broadcast messages injected at root: 0
Fri Mar 27 11:06:12.519 ft: basements decompressed as a target of a query: 1
Fri Mar 27 11:06:12.519 ft: basements decompressed for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: basements decompressed for prefetch: 0
Fri Mar 27 11:06:12.519 ft: basements decompressed for write: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed as a target of a query: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed for prefetch: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed for write: 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for query: 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for query (bytes): 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for query (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: pivots fetched for prefetch: 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: pivots fetched for write: 4
Fri Mar 27 11:06:12.519 ft: pivots fetched for write (bytes): 35840
Fri Mar 27 11:06:12.519 ft: pivots fetched for write (seconds): 0.000009
Fri Mar 27 11:06:12.519 ft: basements fetched as a target of a query: 0
Fri Mar 27 11:06:12.519 ft: basements fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.519 ft: basements fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: basements fetched for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: basements fetched for prefetch: 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: basements fetched for write: 4
Fri Mar 27 11:06:12.519 ft: basements fetched for write (bytes): 35840
Fri Mar 27 11:06:12.519 ft: basements fetched for write (seconds): 0.000002
Fri Mar 27 11:06:12.519 ft: buffers fetched as a target of a query: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: buffers fetched for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: buffers fetched for prefetch: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: buffers fetched for write: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for write (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for write (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: leaf compression to memory (seconds): 0.002101
Fri Mar 27 11:06:12.519 ft: leaf serialization to memory (seconds): 0.000085
Fri Mar 27 11:06:12.519 ft: leaf decompression to memory (seconds): 0.000469
Fri Mar 27 11:06:12.519 ft: leaf deserialization to memory (seconds): 0.000131
Fri Mar 27 11:06:12.519 ft: nonleaf compression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: nonleaf serialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: nonleaf decompression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: nonleaf deserialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: promotion: roots split: 0
Fri Mar 27 11:06:12.519 ft: promotion: leaf roots injected into: 11
Fri Mar 27 11:06:12.519 ft: promotion: h1 roots injected into: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 0: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 1: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 2: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 3: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections lower than depth 3: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped because of a nonempty buffer: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped at height 1: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped because the child was locked or not at all in memory: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped because the child was not fully in memory: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped anyway, after locking the child: 0
Fri Mar 27 11:06:12.519 ft: basement nodes deserialized with fixed-keysize: 5
Fri Mar 27 11:06:12.519 ft: basement nodes deserialized with variable-keysize: 8
Fri Mar 27 11:06:12.519 ft: promotion: succeeded in using the rightmost leaf shortcut: 0
Fri Mar 27 11:06:12.519 ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 0
Fri Mar 27 11:06:12.519 ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 0
Fri Mar 27 11:06:12.519 ft flusher: total nodes potentially flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: height-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: height-greater-than-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: nodes cleaned which had empty buffers: 0
Fri Mar 27 11:06:12.519 ft flusher: nodes dirtied by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: max bytes in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: min bytes in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.519 ft flusher: total bytes in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: max workdone in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: min workdone in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.519 ft flusher: total workdone in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: times cleaner thread tries to merge a leaf: 0
Fri Mar 27 11:06:12.519 ft flusher: cleaner thread leaf merges in progress: 0
Fri Mar 27 11:06:12.519 ft flusher: cleaner thread leaf merges successful: 0
Fri Mar 27 11:06:12.519 ft flusher: nodes dirtied by cleaner thread leaf merges: 0
Fri Mar 27 11:06:12.519 ft flusher: total number of flushes done by flusher threads or cleaner threads: 0
Fri Mar 27 11:06:12.519 ft flusher: number of in memory flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that read something off disk: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered another flush in child: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 1 cascading flush: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 2 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 3 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 4 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 5 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered over 5 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: leaf node splits: 0
Fri Mar 27 11:06:12.519 ft flusher: nonleaf node splits: 0
Fri Mar 27 11:06:12.519 ft flusher: leaf node merges: 0
Fri Mar 27 11:06:12.519 ft flusher: nonleaf node merges: 0
Fri Mar 27 11:06:12.519 ft flusher: leaf node balances: 0
Fri Mar 27 11:06:12.519 hot: operations ever started: 0
Fri Mar 27 11:06:12.519 hot: operations successfully completed: 0
Fri Mar 27 11:06:12.519 hot: operations aborted: 0
Fri Mar 27 11:06:12.519 hot: max number of flushes from root ever required to optimize a tree: 0
Fri Mar 27 11:06:12.519 txn: begin: 695
Fri Mar 27 11:06:12.519 txn: begin read only: 2
Fri Mar 27 11:06:12.519 txn: successful commits: 453
Fri Mar 27 11:06:12.519 txn: aborts: 242
Fri Mar 27 11:06:12.519 logger: next LSN: 2565864190
Fri Mar 27 11:06:12.519 logger: writes: 6
Fri Mar 27 11:06:12.519 logger: writes (bytes): 25888
Fri Mar 27 11:06:12.519 logger: writes (uncompressed bytes): 25888
Fri Mar 27 11:06:12.519 logger: writes (seconds): 0.000031
Fri Mar 27 11:06:12.519 logger: number of long logger write operations: 0
Fri Mar 27 11:06:12.519 indexer: number of indexers successfully created: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to toku_indexer_create_indexer() that failed: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->build() succeeded: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->build() failed: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->close() that succeeded: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->close() that failed: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->abort(): 0
Fri Mar 27 11:06:12.519 indexer: number of indexers currently in existence: 0
Fri Mar 27 11:06:12.519 indexer: max number of indexers that ever existed simultaneously: 0
Fri Mar 27 11:06:12.519 loader: number of loaders successfully created: 0
Fri Mar 27 11:06:12.519 loader: number of calls to toku_loader_create_loader() that failed: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->put() succeeded: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->put() failed: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->close() that succeeded: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->close() that failed: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->abort(): 0
Fri Mar 27 11:06:12.519 loader: number of loaders currently in existence: 0
Fri Mar 27 11:06:12.519 loader: max number of loaders that ever existed simultaneously: 0
Fri Mar 27 11:06:12.519 memory: number of malloc operations: 0
Fri Mar 27 11:06:12.519 memory: number of free operations: 0
Fri Mar 27 11:06:12.519 memory: number of realloc operations: 0
Fri Mar 27 11:06:12.519 memory: number of malloc operations that failed: 0
Fri Mar 27 11:06:12.519 memory: number of realloc operations that failed: 0
Fri Mar 27 11:06:12.519 memory: number of bytes requested: 0
Fri Mar 27 11:06:12.519 memory: number of bytes used (requested + overhead): 0
Fri Mar 27 11:06:12.519 memory: number of bytes freed: 0
Fri Mar 27 11:06:12.519 memory: largest attempted allocation size: 0
Fri Mar 27 11:06:12.519 memory: size of the last failed allocation attempt: 0
Fri Mar 27 11:06:12.519 memory: estimated maximum memory footprint: 0
Fri Mar 27 11:06:12.519 memory: mallocator version: tokumx-2.0.0-2-0-gc7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.519 memory: mmap threshold: 4194304
Fri Mar 27 11:06:12.519 filesystem: ENOSPC redzone state: 0
Fri Mar 27 11:06:12.519 filesystem: threads currently blocked by full disk: 0
Fri Mar 27 11:06:12.519 filesystem: number of operations rejected by enospc prevention (red zone): 0
Fri Mar 27 11:06:12.519 filesystem: most recent disk full: Thu Jan 1 01:00:00 1970
Fri Mar 27 11:06:12.519 filesystem: number of write operations that returned ENOSPC: 0
Fri Mar 27 11:06:12.519 filesystem: fsync time: 554923
Fri Mar 27 11:06:12.519 filesystem: fsync count: 471
Fri Mar 27 11:06:12.519 filesystem: long fsync time: 0
Fri Mar 27 11:06:12.519 filesystem: long fsync count: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a full fetch: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a partial fetch: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a full eviction: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a partial eviction: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a message injection: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a message application: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a flush: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a the cleaner thread: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a full fetch (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a partial fetch (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a full eviction (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a partial eviction (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a message injection: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a message application: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a flush: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by the cleaner thread: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.519 context: something uninstrumented blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.519 Memory usage:
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.525 Crash reason:
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 terminate() called
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.525 Process info:
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 OS: Linux Ubuntu 14.04 x86_64
Fri Mar 27 11:06:12.525 NCPU: 8
Fri Mar 27 11:06:12.526 VIRT: 518 MB
Fri Mar 27 11:06:12.526 RES: 518 MB
Fri Mar 27 11:06:12.526 PHYS: 32039 MB
Fri Mar 27 11:06:12.526 RLIMIT_CORE: 0 (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_NOFILE: 1024 (soft), 4096 (hard)
Fri Mar 27 11:06:12.526 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_AS: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 _SC_OPEN_MAX: 1024
Fri Mar 27 11:06:12.526 _SC_PAGESIZE: 4096
Fri Mar 27 11:06:12.526 _SC_PHYS_PAGES: 8202028
Fri Mar 27 11:06:12.526 _SC_AVPHYS_PAGES: 4375019
Fri Mar 27 11:06:12.526 _SC_NPROCESSORS_CONF: 8
Fri Mar 27 11:06:12.526 _SC_NPROCESSORS_ONLN: 8
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Parsed server options:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 dbpath: "atk2/tokumx-atk2.bak/"
Fri Mar 27 11:06:12.526 port: 13456
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Filesystem information:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 Information for dbpath "/srv/atk2/tokumx-atk2.bak/":
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 type magic: 0x9123683E
Fri Mar 27 11:06:12.526 type: unknown
Fri Mar 27 11:06:12.526 bsize: 4096
Fri Mar 27 11:06:12.526 blocks: 947132928
Fri Mar 27 11:06:12.526 bfree: 606012620
Fri Mar 27 11:06:12.526 bavail: 605308364
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Current operations in progress:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 { opid: 2, active: true, secs_running: 0, op: "query", ns: "maindb.system.namespaces", query: { listDatabases: 1.0 }, client: "127.0.0.1:57521", desc: "conn1", threadId: "0x7f05ad501700", connectionId: 1, rootTxnid: 631617053, locks: { ^: "R" }, waitingForLock: false, lockStats: { timeLockedMicros: { r: 67, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "websvr", threadId: "0x7f058e7ff700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "clientcursormon", threadId: "0x7f0590bfb700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "snapshotthread", threadId: "0x7f05913fc700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "insert", ns: "local.startup_log", insert: {}, client: "0.0.0.0:0", desc: "initandlisten", threadId: "0x7f05ad50a840", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 87, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 OpDebug info:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 query admin.$cmd query: { listDatabases: 1.0 } ntoreturn:1 keyUpdates:0 locks(micros) r:67 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 locks(micros) r:87 0ms
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 ================================================================================
Fri Mar 27 11:06:12.526 Fatal error detected
Fri Mar 27 11:06:12.526 ================================================================================
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 About to gather debugging information, please include all of the following along
Fri Mar 27 11:06:12.526 with logs from other servers in the cluster in a bug report.
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Version info:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 tokumxVersion: 2.0.0
Fri Mar 27 11:06:12.526 gitVersion: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.526 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 11:06:12.526 sysInfo: Linux c1ad9bbe58cd 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Fri Mar 27 11:06:12.526 loaderFlags:
Fri Mar 27 11:06:12.526 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local-typedefs -fno-builtin-memcmp -O3
Fri Mar 27 11:06:12.526 debug: false
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Simple stacktrace:
Fri Mar 27 11:06:12.526
0xc4efe3 0x953177 0x955fbb 0x776c97 0x7f05ab7efd40 0x7f05ab7efcc9 0x7f05ab7f30d8 0x776905 0x7f05ac0f9836 0x7f05ac0f88f9 0x7f05ac0f94aa 0x7f05abb8eff3 0x7f05abb8f517 0x86b3a6 0x86b3d5 0x967a31 0x980974 0x980b1d 0xa090bc 0xa2dccf
/opt/tokumx/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4efe3]
/opt/tokumx/bin/mongod() [0x953177]
/opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
/opt/tokumx/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x776c97]
/lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f05ab7efd40]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f05ab7efcc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f05ab7f30d8]
/opt/tokumx/bin/mongod() [0x776905]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
/lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
/opt/tokumx/bin/mongod() [0x86b3a6]
/opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
/opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
/opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
/opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
Fri Mar 27 11:06:12.529
Fri Mar 27 11:06:12.529 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.529 TokuKV engine backtrace:
Fri Mar 27 11:06:12.529
Fri Mar 27 11:06:12.529 Backtrace: (Note: toku_do_assert=0x0x7f05acbe68b0)
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/../lib64/libtokuportability.so(_Z27db_env_do_backtrace_errfuncPFvPKviPKczES0_+0x1f) [0x7f05acbe7c9f]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/../lib64/libtokufractaltree.so(+0x1437d) [0x7f05ace0137d]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod() [0x953248]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x776c97]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f05ab7efd40]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f05ab7efcc9]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f05ab7f30d8]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod() [0x776905]
Fri Mar 27 11:06:12.535 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
Fri Mar 27 11:06:12.535 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
Fri Mar 27 11:06:12.535 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod() [0x86b3a6]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x39) [0xa1b829]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcd7) [0xa1efc7]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo12_runCommandsEPKcRKNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3aa) [0xa1f8ca]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x962c35]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x6e3) [0x9663b3]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6cb) [0x993a5b]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x8b) [0x78cd3b]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x3dc) [0x89649c]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f05ac7c8182]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f05ab8b400d]
Fri Mar 27 11:06:12.535 BUILD_ID = 107540753
Fri Mar 27 11:06:12.535 time of environment creation: Tue Jan 13 07:26:51 2015
Fri Mar 27 11:06:12.535 time of engine startup: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 time now: Fri Mar 27 11:06:12 2015
Fri Mar 27 11:06:12.535 db opens: 440
Fri Mar 27 11:06:12.535 db closes: 3
Fri Mar 27 11:06:12.535 num open dbs now: 437
Fri Mar 27 11:06:12.535 max open dbs: 440
Fri Mar 27 11:06:12.535 period, in ms, that recovery log is automatically fsynced: 100
Fri Mar 27 11:06:12.535 dictionary inserts: 0
Fri Mar 27 11:06:12.535 dictionary inserts fail: 0
Fri Mar 27 11:06:12.535 dictionary deletes: 1
Fri Mar 27 11:06:12.535 dictionary deletes fail: 0
Fri Mar 27 11:06:12.535 dictionary updates: 0
Fri Mar 27 11:06:12.535 dictionary updates fail: 0
Fri Mar 27 11:06:12.535 dictionary broadcast updates: 0
Fri Mar 27 11:06:12.535 dictionary broadcast updates fail: 0
Fri Mar 27 11:06:12.535 dictionary multi inserts: 1
Fri Mar 27 11:06:12.535 dictionary multi inserts fail: 0
Fri Mar 27 11:06:12.535 dictionary multi deletes: 2
Fri Mar 27 11:06:12.535 dictionary multi deletes fail: 0
Fri Mar 27 11:06:12.535 dictionary updates multi: 0
Fri Mar 27 11:06:12.535 dictionary updates multi fail: 0
Fri Mar 27 11:06:12.535 le: max committed xr: 1
Fri Mar 27 11:06:12.535 le: max provisional xr: 2
Fri Mar 27 11:06:12.535 le: expanded: 0
Fri Mar 27 11:06:12.535 le: max memsize: 883
Fri Mar 27 11:06:12.535 le: size of leafentries before garbage collection (during message application): 0
Fri Mar 27 11:06:12.535 le: size of leafentries after garbage collection (during message application): 0
Fri Mar 27 11:06:12.535 le: size of leafentries before garbage collection (outside message application): 0
Fri Mar 27 11:06:12.535 le: size of leafentries after garbage collection (outside message application): 0
Fri Mar 27 11:06:12.535 checkpoint: period: 60
Fri Mar 27 11:06:12.535 checkpoint: footprint: 0
Fri Mar 27 11:06:12.535 checkpoint: last checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 checkpoint: last complete checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 checkpoint: last complete checkpoint ended: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 checkpoint: time spent during checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.535 checkpoint: time spent during last checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.535 checkpoint: last complete checkpoint LSN: 2565863732
Fri Mar 27 11:06:12.535 checkpoint: checkpoints taken : 2
Fri Mar 27 11:06:12.535 checkpoint: checkpoints failed: 0
Fri Mar 27 11:06:12.535 checkpoint: waiters now: 0
Fri Mar 27 11:06:12.535 checkpoint: waiters max: 0
Fri Mar 27 11:06:12.535 checkpoint: non-checkpoint client wait on mo lock: 0
Fri Mar 27 11:06:12.535 checkpoint: non-checkpoint client wait on cs lock: 0
Fri Mar 27 11:06:12.535 checkpoint: checkpoint begin time: 15
Fri Mar 27 11:06:12.535 checkpoint: long checkpoint begin time: 0
Fri Mar 27 11:06:12.535 checkpoint: long checkpoint begin count: 0
Fri Mar 27 11:06:12.535 cachetable: miss: 12
Fri Mar 27 11:06:12.535 cachetable: miss time: 644
Fri Mar 27 11:06:12.535 cachetable: prefetches: 0
Fri Mar 27 11:06:12.535 cachetable: size current: 406407
Fri Mar 27 11:06:12.535 cachetable: size limit: 17334624256
Fri Mar 27 11:06:12.535 cachetable: size writing: 0
Fri Mar 27 11:06:12.535 cachetable: size nonleaf: 0
Fri Mar 27 11:06:12.535 cachetable: size leaf: 403919
Fri Mar 27 11:06:12.535 cachetable: size rollback: 2488
Fri Mar 27 11:06:12.535 cachetable: size cachepressure: 0
Fri Mar 27 11:06:12.535 cachetable: size currently cloned data for checkpoint: 0
Fri Mar 27 11:06:12.535 cachetable: evictions: 0
Fri Mar 27 11:06:12.535 cachetable: cleaner executions: 3
Fri Mar 27 11:06:12.535 cachetable: cleaner period: 2
Fri Mar 27 11:06:12.535 cachetable: cleaner iterations: 5
Fri Mar 27 11:06:12.535 cachetable: number of waits on cache pressure: 0
Fri Mar 27 11:06:12.535 cachetable: time waiting on cache pressure: 0
Fri Mar 27 11:06:12.535 cachetable: number of long waits on cache pressure: 0
Fri Mar 27 11:06:12.535 cachetable: long time waiting on cache pressure: 0
Fri Mar 27 11:06:12.535 locktree: memory size: 576
Fri Mar 27 11:06:12.535 locktree: memory size limit: 1679775334
Fri Mar 27 11:06:12.535 locktree: number of times lock escalation ran: 0
Fri Mar 27 11:06:12.535 locktree: time spent running escalation (seconds): 0.000000
Fri Mar 27 11:06:12.535 locktree: latest post-escalation memory size: 0
Fri Mar 27 11:06:12.535 locktree: number of locktrees open now: 440
Fri Mar 27 11:06:12.535 locktree: number of pending lock requests: 0
Fri Mar 27 11:06:12.535 locktree: number of locktrees eligible for the STO: 5
Fri Mar 27 11:06:12.535 locktree: number of times a locktree ended the STO early: 0
Fri Mar 27 11:06:12.535 locktree: time spent ending the STO early (seconds): 0.000000
Fri Mar 27 11:06:12.535 locktree: number of wait locks: 0
Fri Mar 27 11:06:12.535 locktree: time waiting for locks: 0
Fri Mar 27 11:06:12.535 locktree: number of long wait locks: 0
Fri Mar 27 11:06:12.535 locktree: long time waiting for locks: 0
Fri Mar 27 11:06:12.535 locktree: number of lock timeouts: 0
Fri Mar 27 11:06:12.535 locktree: number of waits on lock escalation: 0
Fri Mar 27 11:06:12.535 locktree: time waiting on lock escalation: 0
Fri Mar 27 11:06:12.535 locktree: number of long waits on lock escalation: 0
Fri Mar 27 11:06:12.535 locktree: long time waiting on lock escalation: 0
Fri Mar 27 11:06:12.535 ft: dictionary updates: 0
Fri Mar 27 11:06:12.535 ft: dictionary broadcast updates: 0
Fri Mar 27 11:06:12.535 ft: descriptor set: 0
Fri Mar 27 11:06:12.535 ft: messages ignored by leaf due to msn: 0
Fri Mar 27 11:06:12.535 ft: total search retries due to TRY_AGAIN: 0
Fri Mar 27 11:06:12.535 ft: searches requiring more tries than the height of the tree: 0
Fri Mar 27 11:06:12.535 ft: searches requiring more tries than the height of the tree plus three: 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint): 4
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 35840
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 187286
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.000023
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.535 ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.535 ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.535 ft: nonleaf node partial evictions: 0
Fri Mar 27 11:06:12.535 ft: nonleaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf node partial evictions: 0
Fri Mar 27 11:06:12.535 ft: leaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf node full evictions: 0
Fri Mar 27 11:06:12.535 ft: leaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf node full evictions: 0
Fri Mar 27 11:06:12.535 ft: nonleaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes created: 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes created: 0
Fri Mar 27 11:06:12.535 ft: leaf nodes destroyed: 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes destroyed: 0
Fri Mar 27 11:06:12.535 ft: bytes of messages injected at root (all trees): 0
Fri Mar 27 11:06:12.535 ft: bytes of messages flushed from h1 nodes to leaves: 0
Fri Mar 27 11:06:12.535 ft: bytes of messages currently in trees (estimate): 0
Fri Mar 27 11:06:12.535 ft: messages injected at root: 0
Fri Mar 27 11:06:12.535 ft: broadcast messages injected at root: 0
Fri Mar 27 11:06:12.535 ft: basements decompressed as a target of a query: 1
Fri Mar 27 11:06:12.535 ft: basements decompressed for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: basements decompressed for prefetch: 0
Fri Mar 27 11:06:12.535 ft: basements decompressed for write: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed as a target of a query: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed for prefetch: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed for write: 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for query: 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for query (bytes): 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for query (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: pivots fetched for prefetch: 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: pivots fetched for write: 4
Fri Mar 27 11:06:12.535 ft: pivots fetched for write (bytes): 35840
Fri Mar 27 11:06:12.535 ft: pivots fetched for write (seconds): 0.000009
Fri Mar 27 11:06:12.535 ft: basements fetched as a target of a query: 0
Fri Mar 27 11:06:12.535 ft: basements fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.535 ft: basements fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: basements fetched for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: basements fetched for prefetch: 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: basements fetched for write: 4
Fri Mar 27 11:06:12.535 ft: basements fetched for write (bytes): 35840
Fri Mar 27 11:06:12.535 ft: basements fetched for write (seconds): 0.000002
Fri Mar 27 11:06:12.535 ft: buffers fetched as a target of a query: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: buffers fetched for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: buffers fetched for prefetch: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: buffers fetched for write: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for write (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for write (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: leaf compression to memory (seconds): 0.002101
Fri Mar 27 11:06:12.535 ft: leaf serialization to memory (seconds): 0.000085
Fri Mar 27 11:06:12.535 ft: leaf decompression to memory (seconds): 0.000469
Fri Mar 27 11:06:12.535 ft: leaf deserialization to memory (seconds): 0.000131
Fri Mar 27 11:06:12.535 ft: nonleaf compression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf serialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf decompression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf deserialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: promotion: roots split: 0
Fri Mar 27 11:06:12.535 ft: promotion: leaf roots injected into: 11
Fri Mar 27 11:06:12.535 ft: promotion: h1 roots injected into: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 0: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 1: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 2: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 3: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections lower than depth 3: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped because of a nonempty buffer: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped at height 1: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped because the child was locked or not at all in memory: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped because the child was not fully in memory: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped anyway, after locking the child: 0
Fri Mar 27 11:06:12.535 ft: basement nodes deserialized with fixed-keysize: 5
Fri Mar 27 11:06:12.535 ft: basement nodes deserialized with variable-keysize: 8
Fri Mar 27 11:06:12.535 ft: promotion: succeeded in using the rightmost leaf shortcut: 0
Fri Mar 27 11:06:12.535 ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 0
Fri Mar 27 11:06:12.535 ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 0
Fri Mar 27 11:06:12.535 ft flusher: total nodes potentially flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: height-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: height-greater-than-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: nodes cleaned which had empty buffers: 0
Fri Mar 27 11:06:12.535 ft flusher: nodes dirtied by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: max bytes in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: min bytes in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.535 ft flusher: total bytes in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: max workdone in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: min workdone in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.535 ft flusher: total workdone in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: times cleaner thread tries to merge a leaf: 0
Fri Mar 27 11:06:12.535 ft flusher: cleaner thread leaf merges in progress: 0
Fri Mar 27 11:06:12.535 ft flusher: cleaner thread leaf merges successful: 0
Fri Mar 27 11:06:12.535 ft flusher: nodes dirtied by cleaner thread leaf merges: 0
Fri Mar 27 11:06:12.535 ft flusher: total number of flushes done by flusher threads or cleaner threads: 0
Fri Mar 27 11:06:12.535 ft flusher: number of in memory flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that read something off disk: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered another flush in child: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 1 cascading flush: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 2 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 3 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 4 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 5 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered over 5 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: leaf node splits: 0
Fri Mar 27 11:06:12.535 ft flusher: nonleaf node splits: 0
Fri Mar 27 11:06:12.535 ft flusher: leaf node merges: 0
Fri Mar 27 11:06:12.535 ft flusher: nonleaf node merges: 0
Fri Mar 27 11:06:12.535 ft flusher: leaf node balances: 0
Fri Mar 27 11:06:12.535 hot: operations ever started: 0
Fri Mar 27 11:06:12.535 hot: operations successfully completed: 0
Fri Mar 27 11:06:12.535 hot: operations aborted: 0
Fri Mar 27 11:06:12.535 hot: max number of flushes from root ever required to optimize a tree: 0
Fri Mar 27 11:06:12.535 txn: begin: 695
Fri Mar 27 11:06:12.535 txn: begin read only: 2
Fri Mar 27 11:06:12.535 txn: successful commits: 453
Fri Mar 27 11:06:12.535 txn: aborts: 242
Fri Mar 27 11:06:12.535 logger: next LSN: 2565864190
Fri Mar 27 11:06:12.535 logger: writes: 6
Fri Mar 27 11:06:12.535 logger: writes (bytes): 25888
Fri Mar 27 11:06:12.535 logger: writes (uncompressed bytes): 25888
Fri Mar 27 11:06:12.535 logger: writes (seconds): 0.000031
Fri Mar 27 11:06:12.535 logger: number of long logger write operations: 0
Fri Mar 27 11:06:12.535 indexer: number of indexers successfully created: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to toku_indexer_create_indexer() that failed: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->build() succeeded: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->build() failed: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->close() that succeeded: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->close() that failed: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->abort(): 0
Fri Mar 27 11:06:12.535 indexer: number of indexers currently in existence: 0
Fri Mar 27 11:06:12.535 indexer: max number of indexers that ever existed simultaneously: 0
Fri Mar 27 11:06:12.535 loader: number of loaders successfully created: 0
Fri Mar 27 11:06:12.535 loader: number of calls to toku_loader_create_loader() that failed: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->put() succeeded: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->put() failed: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->close() that succeeded: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->close() that failed: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->abort(): 0
Fri Mar 27 11:06:12.535 loader: number of loaders currently in existence: 0
Fri Mar 27 11:06:12.535 loader: max number of loaders that ever existed simultaneously: 0
Fri Mar 27 11:06:12.535 memory: number of malloc operations: 0
Fri Mar 27 11:06:12.535 memory: number of free operations: 0
Fri Mar 27 11:06:12.536 memory: number of realloc operations: 0
Fri Mar 27 11:06:12.536 memory: number of malloc operations that failed: 0
Fri Mar 27 11:06:12.536 memory: number of realloc operations that failed: 0
Fri Mar 27 11:06:12.536 memory: number of bytes requested: 0
Fri Mar 27 11:06:12.536 memory: number of bytes used (requested + overhead): 0
Fri Mar 27 11:06:12.536 memory: number of bytes freed: 0
Fri Mar 27 11:06:12.536 memory: largest attempted allocation size: 0
Fri Mar 27 11:06:12.536 memory: size of the last failed allocation attempt: 0
Fri Mar 27 11:06:12.536 memory: estimated maximum memory footprint: 0
Fri Mar 27 11:06:12.536 memory: mallocator version: tokumx-2.0.0-2-0-gc7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.536 memory: mmap threshold: 4194304
Fri Mar 27 11:06:12.536 filesystem: ENOSPC redzone state: 0
Fri Mar 27 11:06:12.536 filesystem: threads currently blocked by full disk: 0
Fri Mar 27 11:06:12.536 filesystem: number of operations rejected by enospc prevention (red zone): 0
Fri Mar 27 11:06:12.536 filesystem: most recent disk full: Thu Jan 1 01:00:00 1970
Fri Mar 27 11:06:12.536 filesystem: number of write operations that returned ENOSPC: 0
Fri Mar 27 11:06:12.536 filesystem: fsync time: 554923
Fri Mar 27 11:06:12.536 filesystem: fsync count: 471
Fri Mar 27 11:06:12.536 filesystem: long fsync time: 0
Fri Mar 27 11:06:12.536 filesystem: long fsync count: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a full fetch: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a partial fetch: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a full eviction: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a partial eviction: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a message injection: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a message application: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a flush: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a the cleaner thread: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a full fetch (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a partial fetch (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a full eviction (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a partial eviction (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a message injection: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a message application: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a flush: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by the cleaner thread: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.536 context: something uninstrumented blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.536 Memory usage:
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.542 Crash reason:
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 Got signal: 6 (Aborted).
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.542 Process info:
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 OS: Linux Ubuntu 14.04 x86_64
Fri Mar 27 11:06:12.542 NCPU: 8
Fri Mar 27 11:06:12.542 VIRT: 518 MB
Fri Mar 27 11:06:12.542 RES: 518 MB
Fri Mar 27 11:06:12.542 PHYS: 32039 MB
Fri Mar 27 11:06:12.542 RLIMIT_CORE: 0 (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_NOFILE: 1024 (soft), 4096 (hard)
Fri Mar 27 11:06:12.542 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Fri Mar 27 11:06:12.543 RLIMIT_AS: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.543 _SC_OPEN_MAX: 1024
Fri Mar 27 11:06:12.543 _SC_PAGESIZE: 4096
Fri Mar 27 11:06:12.543 _SC_PHYS_PAGES: 8202028
Fri Mar 27 11:06:12.543 _SC_AVPHYS_PAGES: 4375019
Fri Mar 27 11:06:12.543 _SC_NPROCESSORS_CONF: 8
Fri Mar 27 11:06:12.543 _SC_NPROCESSORS_ONLN: 8
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 Parsed server options:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 dbpath: "atk2/tokumx-atk2.bak/"
Fri Mar 27 11:06:12.543 port: 13456
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 Filesystem information:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 Information for dbpath "/srv/atk2/tokumx-atk2.bak/":
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 type magic: 0x9123683E
Fri Mar 27 11:06:12.543 type: unknown
Fri Mar 27 11:06:12.543 bsize: 4096
Fri Mar 27 11:06:12.543 blocks: 947132928
Fri Mar 27 11:06:12.543 bfree: 606012620
Fri Mar 27 11:06:12.543 bavail: 605308364
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 Current operations in progress:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 { opid: 2, active: true, secs_running: 0, op: "query", ns: "maindb.system.namespaces", query: { listDatabases: 1.0 }, client: "127.0.0.1:57521", desc: "conn1", threadId: "0x7f05ad501700", connectionId: 1, rootTxnid: 631617053, locks: { ^: "R" }, waitingForLock: false, lockStats: { timeLockedMicros: { r: 67, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "websvr", threadId: "0x7f058e7ff700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "clientcursormon", threadId: "0x7f0590bfb700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "snapshotthread", threadId: "0x7f05913fc700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "insert", ns: "local.startup_log", insert: {}, client: "0.0.0.0:0", desc: "initandlisten", threadId: "0x7f05ad50a840", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 87, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 OpDebug info:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 query admin.$cmd query: { listDatabases: 1.0 } ntoreturn:1 keyUpdates:0 locks(micros) r:67 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 locks(micros) r:87 0ms
Fri Mar 27 11:06:12.543
We have a replica set with 2 replicas + arbiter. This morning, one of the replicas crashed (and won't start again, will file a separate issue), and the other one works mostly fine, but I can't rebuild the replica set because
listDatabasesmakes the thing crash with [1];<rant>even more awesomer, there is absolutely no repair option I can find, because this code can't ever be wrong and filesystems can't ever get corrupted...</rant>
[1]