Tokumx 2.0 upgrade crashes after startup on some queries

[dazoot]

Tue Sep 30 11:34:28.601 [conn8] Assertion: 13111:field not found, expected type 16 0xc4cfb3 0xacd4f3 0xacd58c 0x7bd172 0x8827b8 0x87dd4b 0x7f0b66ce0444 0x7f0b66d21337 0x7f0b66d21ecc 0x7f0b66d2213c 0x7f0b66d220db 0x7f0b66d22086 0x7f0b66d220db /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4cfb3] /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x93) [0xacd4f3] /usr/bin/mongod() [0xacd58c] /usr/bin/mongod(_ZNK5mongo11BSONElement3chkEi+0x1a2) [0x7bd172] /usr/bin/mongod(_ZNK5mongo11BSONElement3IntEv+0x18) [0x8827b8] /usr/bin/mongod() [0x87dd4b] /usr/lib/tokumx/libtokufractaltree.so(+0x4b444) [0x7f0b66ce0444] /usr/lib/tokumx/libtokufractaltree.so(+0x8c337) [0x7f0b66d21337] /usr/lib/tokumx/libtokufractaltree.so(+0x8cecc) [0x7f0b66d21ecc] /usr/lib/tokumx/libtokufractaltree.so(+0x8d13c) [0x7f0b66d2213c] /usr/lib/tokumx/libtokufractaltree.so(+0x8d0db) [0x7f0b66d220db] /usr/lib/tokumx/libtokufractaltree.so(+0x8d086) [0x7f0b66d22086] /usr/lib/tokumx/libtokufractaltree.so(+0x8d0db) [0x7f0b66d220db] Tue Sep 30 11:34:28.607 [conn8] live.abonat_61 Caught exception in ydb update callback, ex: field not found, expected type 16key: { _id: 42237380 }oldObj: { _id: 42237380, abonat_id: 42237380, abonat_stars: 1.0, active_status: 1, bounce_status: 0, confirm_date: null, domain: "rediffmail.com", email: "test@example.com", firstname: null, last_activity: { date: new Date(1406902502000), type: "view" }, lastname: null, list_id: 324, locations: { view: [ { code: "IN", name: "INDIA", region: "MAHARASHTRA", city: "MUMBAI" }, { code: "IN", name: "INDIA", region: "MAHARASHTRA", city: "MUMBAI" }, { code: "IN", name: "INDIA", region: "MAHARASHTRA", city: "MUMBAI" }, { code: "IN", name: "INDIA", region: "MAHARASHTRA", city: "MUMBAI" }, { code: "IN", name: "INDIA", region: "MAHARASHTRA", city: "MUMBAI" } ] }, props: { country: "null", sex: "M" }, segments: [ 18352, 18372, 19890, 21247, 31484, 31522 ], sub_date: "2014-03-17", sub_ip: "86.125.58.125", sub_time: "13:35:41", sub_type: "import", subscription_status: 1, viewed_newsletters: [ 72313, 72851, 73590, 75126, 75963, 77227, 77771, 78537, 78905, 79737, 80171, 80732, 81154, 97872 ] }msg: { t: "u", o: { $unset: { abonat_stars: true } } } Tue Sep 30 11:34:28.607 [conn8] live.abonat_61 Fatal Assertion 17215 0xc4cfb3 0xacd2ec 0x87e267 0x7f0b66ce0444 0x7f0b66d21337 0x7f0b66d21ecc 0x7f0b66d2213c 0x7f0b66d220db 0x7f0b66d22086 0x7f0b66d220db /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4cfb3] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x4c) [0xacd2ec] /usr/bin/mongod() [0x87e267] /usr/lib/tokumx/libtokufractaltree.so(+0x4b444) [0x7f0b66ce0444] /usr/lib/tokumx/libtokufractaltree.so(+0x8c337) [0x7f0b66d21337] /usr/lib/tokumx/libtokufractaltree.so(+0x8cecc) [0x7f0b66d21ecc] /usr/lib/tokumx/libtokufractaltree.so(+0x8d13c) [0x7f0b66d2213c] /usr/lib/tokumx/libtokufractaltree.so(+0x8d0db) [0x7f0b66d220db] /usr/lib/tokumx/libtokufractaltree.so(+0x8d086) [0x7f0b66d22086] /usr/lib/tokumx/libtokufractaltree.so(+0x8d0db) [0x7f0b66d220db] Tue Sep 30 11:34:28.609 [conn8]

***aborting after fassert() failure

Tue Sep 30 11:34:28.609 Tue Sep 30 11:34:28.609 ================================================================================ Tue Sep 30 11:34:28.609 Fatal error detected Tue Sep 30 11:34:28.610 ================================================================================ Tue Sep 30 11:34:28.610 Tue Sep 30 11:34:28.610 About to gather debugging information, please include all of the following along Tue Sep 30 11:34:28.610 with logs from other servers in the cluster in a bug report. Tue Sep 30 11:34:28.610 Tue Sep 30 11:34:28.610 -------------------------------------------------------------------------------- Tue Sep 30 11:34:28.610 Version info: Tue Sep 30 11:34:28.610 Tue Sep 30 11:34:28.610 tokumxVersion: 2.0.0 Tue Sep 30 11:34:28.610 gitVersion: 3b5cfed09fe2464314e7035fe088748ef1910343 Tue Sep 30 11:34:28.610 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83 Tue Sep 30 11:34:28.610 sysInfo: Linux 217b861cf5d7 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49 Tue Sep 30 11:34:28.610 loaderFlags: Tue Sep 30 11:34:28.610 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local-typedefs -fno-builtin-memcmp -O3 Tue Sep 30 11:34:28.610 debug: false Tue Sep 30 11:34:28.610 Tue Sep 30 11:34:28.610 -------------------------------------------------------------------------------- Tue Sep 30 11:34:28.610 Simple stacktrace: Tue Sep 30 11:34:28.610 0xc4cfb3 0x9892d7 0x98c76b 0x7ae377 0x7f0b65529220 0x7f0b655291a5 0x7f0b6552c420 0xacd327 0x87e267 0x7f0b66ce0444 0x7f0b66d21337 0x7f0b66d21ecc 0x7f0b66d2213c 0x7f0b66d220db 0x7f0b66d22086 0x7f0b66d220db /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4cfb3] /usr/bin/mongod() [0x9892d7] /usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x98c76b] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x7ae377] /lib/x86_64-linux-gnu/libc.so.6(+0x32220) [0x7f0b65529220] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f0b655291a5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7f0b6552c420] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x87) [0xacd327] /usr/bin/mongod() [0x87e267] /usr/lib/tokumx/libtokufractaltree.so(+0x4b444) [0x7f0b66ce0444] /usr/lib/tokumx/libtokufractaltree.so(+0x8c337) [0x7f0b66d21337] /usr/lib/tokumx/libtokufractaltree.so(+0x8cecc) [0x7f0b66d21ecc] /usr/lib/tokumx/libtokufractaltree.so(+0x8d13c) [0x7f0b66d2213c] /usr/lib/tokumx/libtokufractaltree.so(+0x8d0db) [0x7f0b66d220db] /usr/lib/tokumx/libtokufractaltree.so(+0x8d086) [0x7f0b66d22086] /usr/lib/tokumx/libtokufractaltree.so(+0x8d0db) [0x7f0b66d220db]

[dazoot]

Also same problem on mongodump:

Tue Sep 30 12:40:12.100 [tools] Collection File Writing Progress: 7047200/8509831 82% (objects) Tue Sep 30 12:40:12.115 Assertion: 13111:field not found, expected type 16 0xef3823 0xb5f853 0xb5f8ec 0x7efbc2 0x88933b 0xa63edb 0x7f7593d3c444 0x7f7593d7d337 0x7f7593d9e54c 0x7f7593d9ea5a 0x7f7593d9fd1b 0x7f7593d3373f 0x7f7593d529fe 0x7f75938cfb50 0x7f7592412e6d mongodump(_ZN5mongo15printStackTraceERSo+0x23) [0xef3823] mongodump(_ZN5mongo11msgassertedEiPKc+0x93) [0xb5f853] mongodump() [0xb5f8ec] mongodump(_ZNK5mongo11BSONElement3chkEi+0x192) [0x7efbc2] mongodump(_ZNK5mongo11BSONElement3IntEv+0xb) [0x88933b] mongodump() [0xa63edb] /usr/lib/tokumx/libtokufractaltree.so(+0x4b444) [0x7f7593d3c444] /usr/lib/tokumx/libtokufractaltree.so(+0x8c337) [0x7f7593d7d337] /usr/lib/tokumx/libtokufractaltree.so(+0xad54c) [0x7f7593d9e54c] /usr/lib/tokumx/libtokufractaltree.so(+0xada5a) [0x7f7593d9ea5a] /usr/lib/tokumx/libtokufractaltree.so(+0xaed1b) [0x7f7593d9fd1b] /usr/lib/tokumx/libtokufractaltree.so(+0x4273f) [0x7f7593d3373f] /usr/lib/tokumx/libtokufractaltree.so(+0x619fe) [0x7f7593d529fe] /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f75938cfb50] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7592412e6d] Tue Sep 30 12:40:12.139 Caught exception in ydb update callback, ex: field not found, expected type 16key: { _id: 690248 }oldObj: { _id: 690248.0, abonat_id: 690248, abonat_stars: 55.0, active_status: 1, bounce_status: 0, confirm_date: null, domain: "example.com", email: "test@example.com", firstname: "Test", history: { 2011-08: [ { action: "view", data: "2011-08-12 14:09", meta: { newsletter_id: 440, subject: "##fullname##: Test" } }, { action: "view", data: "2011-08-24 20:59", meta: { newsletter_id: 444, subject: "##fullname##:" } }, { action: "view", data: "2011-08-31 07:05", meta: { newsletter_id: 453, subject: "##fullname##:" } } ] }, lastname: "Test", list_id: 1, locations: { view: [ { code: "FR", name: "FRANCE", region: "ILE-DE-FRANCE", city: "PARIS" }, { code: "FR", name: "FRANCE", region: "ILE-DE-FRANCE", city: "PARIS" }, { code: "FR", name: "FRANCE", region: "ILE-DE-FRANCE", city: "PARIS" }, { code: "FR", name: "FRANCE", region: "ILE-DE-FRANCE", city: "PARIS" }, { code: "FR", name: "FRANCE", region: "ILE-DE-FRANCE", city: "PARIS" } ] }, props: { is_tx: true, lang: "fr", register_date: "2010-10-12 15:24:57" }, segments: [ 9.0 ], sent_newsletters: {}, sub_date: "2010-10-15", sub_ip: null, sub_time: "15:40:05", sub_type: "import", subscription_status: 1, viewed_newsletters: [ 722, 754, 767, 776, 830 ] }msg: { t: "u", o: { $pull: { sent_newsletters: { $lte: 888 }, viewed_newsletters: { $lte: 888 }, clicked_newsletters: { $lte: 888 } }, $unset: { history.2013-12: 1, history.2014-01: 1, history.2014-02: 1, history.2014-03: 1, history.2014-04: 1, history.2014-05: 1, history.2014-06: 1 } } } Tue Sep 30 12:40:12.139 Fatal Assertion 17215 0xef3823 0xb5f64c 0xa643f7 0x7f7593d3c444 0x7f7593d7d337 0x7f7593d9e54c 0x7f7593d9ea5a 0x7f7593d9fd1b 0x7f7593d3373f 0x7f7593d529fe 0x7f75938cfb50 0x7f7592412e6d mongodump(_ZN5mongo15printStackTraceERSo+0x23) [0xef3823] mongodump(_ZN5mongo13fassertFailedEi+0x4c) [0xb5f64c] mongodump() [0xa643f7] /usr/lib/tokumx/libtokufractaltree.so(+0x4b444) [0x7f7593d3c444] /usr/lib/tokumx/libtokufractaltree.so(+0x8c337) [0x7f7593d7d337] /usr/lib/tokumx/libtokufractaltree.so(+0xad54c) [0x7f7593d9e54c] /usr/lib/tokumx/libtokufractaltree.so(+0xada5a) [0x7f7593d9ea5a] /usr/lib/tokumx/libtokufractaltree.so(+0xaed1b) [0x7f7593d9fd1b] /usr/lib/tokumx/libtokufractaltree.so(+0x4273f) [0x7f7593d3373f] /usr/lib/tokumx/libtokufractaltree.so(+0x619fe) [0x7f7593d529fe] /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f75938cfb50] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7592412e6d] Tue Sep 30 12:40:12.140

***aborting after fassert() failure

[zkasheff]

We've moved over to jira for reporting the issue, can you give more information? It seems that an update that previously happened is now failing to get applied. Did you have fast updates enabled when you were running updates? Can you provide your data directory for us to examine?

[leifwalsh]

When you make a jira ticket, please also describe how you got this build. The stacks look wrong for one of our builds, did you build it yourself? Can you reproduce with an official build?

[dazoot]

I got the build via apt-get ugrade. Opening JIRA ticket right now.

[dazoot]

Reported in JIRA: https://tokutek.atlassian.net/browse/MX-1302

[leifwalsh]

thanks