search

manticoresoftware / manticoresearch

Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
https://manticoresearch.com
GNU General Public License v3.0
8.87k stars 490 forks source link

CRASH DUMP while loading big data #1097

Open utkarsh-t opened 1 year ago

utkarsh-t commented 1 year ago

Describe the bug Whenever i am trying to select result of big data with each index being of size 3GB+ , i am unable to load the table data in multiple tables,

To Reproduce Steps to reproduce the behavior:

  1. running mysql server
  2. SELECT * FROM rule_logs_2023_01_13,rule_logs_2023_01_14,rule_logs_2023_01_27, rule_logs_2023_02_10,rule_logs_2023_02_11 ORDER BY data.timestamp DESC LIMIT 0,30 OPTION max_matches=33

Expected behavior all the table data satisfying query should appear

Describe the environment:

Messages from log files: [Fri Apr 14 08:47:38.556 2023] [14252] preread 12 tables in 77.768 sec ------- FATAL: CRASH DUMP ------- [Fri Apr 14 08:59:26.104 2023] [14249]

--- crashed SphinxQL request dump --- SELECT * FROM rule_logs_2023_01_13,rule_logs_2023_01_14,rule_logs_2023_01_27, rule_logs_2023_02_10,rule_logs_2023_02_11 ORDER BY data.timestamp DESC LIMIT 0,30 OPTION max_matches=33 --- request dump end --- --- local index:rule_logs_2023_02_10 Manticore 6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306) Handling signal 11 -------------- backtrace begins here --------------- Program compiled with Clang 15.0.4 Configured with flags: Configured with these definitions: -DDISTR_BUILD=buster -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore Built on Linux x86_64 (buster) (cross-compiled) Stack bottom = 0x7f91780551b0, thread stack size = 0x20000 Trying manual backtrace: Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1) Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f9178060000, stacksize=0x20000) Trying system backtrace: begin of system symbols: /usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55b46947f17a] /usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55b469342765] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f917e453730] /usr/bin/searchd(_Z16sphJsonFindByKey12ESphJsonTypePPKhPKvij+0x89)[0x55b469759889] /usr/bin/searchd(_ZNK18Expr_JsonFastKey_c9Int64EvalERK9CSphMatch+0x48)[0x55b4696d1388] /usr/bin/searchd(_ZNK24ExprSortJson2StringPtr_c10StringEvalERK9CSphMatchPPKh+0x2a)[0x55b4694a202a] /usr/bin/searchd(_ZNK8ISphExpr16StringEvalPackedERK9CSphMatch+0x20)[0x55b4696aa950] /usr/bin/searchd(_Z15CalcContextItemR9CSphMatchRKN16CSphQueryContext10CalcItem_tE+0x113)[0x55b469443f03] /usr/bin/searchd(+0x8bc5eb)[0x55b4694085eb] /usr/bin/searchd(_ZNK13CSphIndex_VLN18RunFullscanOnAttrsERK17RowIdBoundaries_tRK16CSphQueryContextR19CSphQueryResultMetaRK11VecTraits_TIP15ISphMatchSorterER9CSphMatchibil+0x8d2)[0x55b4693dd752] /usr/bin/searchd(_ZNK13CSphIndex_VLN9MultiScanER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgsl+0x9d7)[0x55b4693e18f7] /usr/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs+0x2bc)[0x55b4693ebecc] /usr/bin/searchd(+0xbfd11e)[0x55b46974911e] /usr/bin/searchd(+0xcffd8d)[0x55b46984bd8d] /usr/bin/searchd(_ZN7Threads4Coro8ExecuteNEiOSt8functionIFvvEE+0x74)[0x55b469c47b34] /usr/bin/searchd(_ZNK9RtIndex_c10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs+0xfc7)[0x55b469738347] /usr/bin/searchd(_ZNK13CSphIndexStub12MultiQueryExEiPK9CSphQueryP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x71)[0x55b46944bc51] /usr/bin/searchd(+0x857a7b)[0x55b4693a3a7b] /usr/bin/searchd(+0xcffc12)[0x55b46984bc12] /usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8_invokeES9+0x1c)[0x55b469c4a64c] /usr/bin/searchd(make_fcontext+0x2f)[0x55b469c6936f] Trying boost backtrace: 0# sphBacktrace(int, bool) in /usr/bin/searchd 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd 2# 0x00007F917E453730 in /lib/x86_64-linux-gnu/libpthread.so.0 3# sphJsonFindByKey(ESphJsonType, unsigned char const*, void const, int, unsigned int) in /usr/bin/searchd 4# Expr_JsonFastKey_c::Int64Eval(CSphMatch const&) const in /usr/bin/searchd 5# ExprSortJson2StringPtr_c::StringEval(CSphMatch const&, unsigned char const) const in /usr/bin/searchd 6# ISphExpr::StringEvalPacked(CSphMatch const&) const in /usr/bin/searchd 7# CalcContextItem(CSphMatch&, CSphQueryContext::CalcItem_t const&) in /usr/bin/searchd 8# 0x000055B4694085EB in /usr/bin/searchd 9# CSphIndex_VLN::RunFullscanOnAttrs(RowIdBoundaries_t const&, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter> const&, CSphMatch&, int, bool, int, long) const in /usr/bin/searchd 10# CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter> const&, CSphMultiQueryArgs const&, long) const in /usr/bin/searchd 11# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter> const&, CSphMultiQueryArgs const&) const in /usr/bin/searchd 12# 0x000055B46974911E in /usr/bin/searchd 13# 0x000055B46984BD8D in /usr/bin/searchd 14# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in /usr/bin/searchd 15# RtIndex_c::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter> const&, CSphMultiQueryArgs const&) const in /usr/bin/searchd 16# CSphIndexStub::MultiQueryEx(int, CSphQuery const, CSphQueryResult, ISphMatchSorter, CSphMultiQueryArgs const&) const in /usr/bin/searchd 17# 0x000055B4693A3A7B in /usr/bin/searchd 18# 0x000055B46984BC12 in /usr/bin/searchd 19# Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd 20# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here --------------- Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues) and attach there: a) searchd log, b) searchd binary, c) searchd symbols. Look into the chapter 'Reporting bugs' in the manual (https://manual.manticoresearch.com/Reporting_bugs) Dump with GDB via watchdog --- active threads --- thd 0 (work_1), proto mysql, state query, command select thd 1 (work_2), proto mysql, state query, command select thd 2 (work_3), proto mysql, state query, command select --- Totally 4 threads, and 3 client-working threads --- ------- CRASH DUMP END ------- [Fri Apr 14 08:59:37.256 2023] [25807] watchdog: main process 14249 crashed via CRASH_EXIT (exit code 2), will be restarted [Fri Apr 14 08:59:37.284 2023] [25807] watchdog: main process 14890 forked ok [Fri Apr 14 08:59:37.548 2023] [14890] starting daemon version '6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306)' ... [Fri Apr 14 08:59:37.548 2023] [14890] listening on all interfaces for sphinx and http(s), port=9312 [Fri Apr 14 08:59:37.548 2023] [14890] listening on all interfaces for mysql, port=9306 [Fri Apr 14 08:59:37.548 2023] [14890] listening on all interfaces for sphinx and http(s), port=9308 [Fri Apr 14 08:59:43.468 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.001 [Fri Apr 14 08:59:43.527 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.527 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.001; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.527 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.002 [Fri Apr 14 08:59:43.527 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.527 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.002; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.527 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.003 [Fri Apr 14 08:59:43.540 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.540 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.003; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.540 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.004 [Fri Apr 14 08:59:43.540 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.540 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.004; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.540 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.005 [Fri Apr 14 08:59:43.550 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.550 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.005; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.550 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.006 [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.551 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.006; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.007 [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.551 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.007; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.008 [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.551 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.008; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.009 [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.551 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.009; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.551 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.010 [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.552 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.010; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.011 [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.552 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.011; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.012 [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.552 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.012; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.013 [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.552 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.013; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.552 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.014 [Fri Apr 14 08:59:43.553 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.553 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.014; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.553 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.015 [Fri Apr 14 08:59:43.553 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.553 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.015; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.553 2023] [14892] binlog: replaying log /var/log/manticoredata/binlog/binlog.016 [Fri Apr 14 08:59:43.553 2023] [14892] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables [Fri Apr 14 08:59:43.553 2023] [14892] binlog: finished replaying /var/log/manticoredata/binlog/binlog.016; 0.0 MB in 0.000 sec [Fri Apr 14 08:59:43.553 2023] [14892] binlog: finished replaying total 16 in 0.085 sec [Fri Apr 14 08:59:43.563 2023] [14891] prereading 12 tables [Fri Apr 14 08:59:53.604 2023] [14890] accepting connections [Fri Apr 14 08:59:56.612 2023] [14893] WARNING: [BUDDY] failed to start after 3 sec [Fri Apr 14 09:01:05.261 2023] [14891] preread 12 tables in 81.699 sec

Additional context Add any other context about the problem here. In case you've faced a crash what indextool --check returns.

sanikolaev commented 1 year ago

Please check if the issue persists in the latest dev version - https://mnt.cr/nightly IIRC we fixed a similar issue recently.

tomatolog commented 1 year ago

could you also check your data there crash happen with indextool ?

indextool --check rule_logs_2023_02_10
utkarsh-t commented 1 year ago

Please check if the issue persists in the latest dev version - https://mnt.cr/nightly IIRC we fixed a similar issue recently.

Yes , iam using the latest version of manticore 6.0.4

utkarsh-t commented 1 year ago

indextool --check rule_logs_2023_02_10

yes i am getting this response WARNING: failed to load RAM chunks, checking only 4 disk chunks checking schema... checking disk chunk, extension 0, 0(4)... checking schema... checking dictionary... checking data... checking rows... checking attribute blocks index... checking kill-list... checking dead row map... checking doc-id lookup... check passed, 0.3 sec elapsed checking disk chunk, extension 1, 1(4)... checking schema... checking dictionary... checking data... checking rows... checking attribute blocks index... checking kill-list... checking dead row map... checking doc-id lookup... check passed, 1.0 sec elapsed checking disk chunk, extension 2, 2(4)... checking schema... checking dictionary... checking data... checking rows... FAILED, Unknown blob row type: 110 at offset 28244406, docid=4469609979696011195, rowid=62032 of 241500 FAILED, Unknown blob row type: 97 at offset 28244884, docid=4469609979696011196, rowid=62033 of 241500 FAILED, Unknown blob row type: 62 at offset 28255239, docid=4469609979696011219, rowid=62056 of 241500 FAILED, Unknown blob row type: 7 at offset 28255481, docid=4469609979696011220, rowid=62057 of 241500 FAILED, Unknown blob row type: 62 at offset 28255717, docid=4469609979696011221, rowid=62058 of 241500 FAILED, Unknown blob row type: 152 at offset 28255956, docid=4469609979696011222, rowid=62059 of 241500 FAILED, Unknown blob row type: 62 at offset 28256821, docid=4469609979696011224, rowid=62061 of 241500 FAILED, Unknown blob row type: 116 at offset 28257875, docid=4469609979696011227, rowid=62064 of 241500 FAILED, Unknown blob row type: 7 at offset 28258342, docid=4469609979696011228, rowid=62065 of 241500 FAILED, Unknown blob row type: 7 at offset 28259062, docid=4469609979696011230, rowid=62067 of 241500 FAILED, Unknown blob row type: 7 at offset 28259782, docid=4469609979696011232, rowid=62069 of 241500 FAILED, Unknown blob row type: 116 at offset 28260275, docid=4469609979696011233, rowid=62070 of 241500 FAILED, Unknown blob row type: 197 at offset 28261209, docid=4469609979696011235, rowid=62072 of 241500 FAILED, Unknown blob row type: 34 at offset 28262409, docid=4469609979696011238, rowid=62075 of 241500 FAILED, Unknown blob row type: 185 at offset 28263176, docid=4469609979696011240, rowid=62077 of 241500 FAILED, Unknown blob row type: 82 at offset 28264178, docid=4469609979696011243, rowid=62080 of 241500 FAILED, Unknown blob row type: 191 at offset 28264425, docid=4469609979696011244, rowid=62081 of 241500 FAILED, Unknown blob row type: 140 at offset 28264920, docid=4469609979696011246, rowid=62083 of 241500 FAILED, Unknown blob row type: 127 at offset 28265392, docid=4469609979696011247, rowid=62084 of 241500 FAILED, Unknown blob row type: 64 at offset 28266073, docid=4469609979696011249, rowid=62086 of 241500 FAILED, Unknown blob row type: 62 at offset 28266471, docid=4469609979696011250, rowid=62087 of 241500 FAILED, Unknown blob row type: 116 at offset 28267491, docid=4469609979696011252, rowid=62089 of 241500 FAILED, Unknown blob row type: 127 at offset 28268656, docid=4469609979696011255, rowid=62092 of 241500 FAILED, Unknown blob row type: 62 at offset 28269383, docid=4469609979696011257, rowid=62094 of 241500 FAILED, Unknown blob row type: 127 at offset 28269616, docid=4469609979696011258, rowid=62095 of 241500 FAILED, Unknown blob row type: 7 at offset 28270102, docid=4469609979696011260, rowid=62097 of 241500 FAILED, Unknown blob row type: 83 at offset 28271154, docid=4469609979696011262, rowid=62099 of 241500 FAILED, Unknown blob row type: 152 at offset 28271364, docid=4469609979696008463, rowid=62100 of 241500 FAILED, Unknown blob row type: 64 at offset 28271601, docid=4469609979696008464, rowid=62101 of 241500 FAILED, Unknown blob row type: 116 at offset 28271843, docid=4469609979696008465, rowid=62102 of 241500 FAILED, Unknown blob row type: 62 at offset 28272085, docid=4469609979696008466, rowid=62103 of 241500 FAILED, Unknown blob row type: 6 at offset 28272490, docid=4469609979696008467, rowid=62104 of 241500 FAILED, Unknown blob row type: 191 at offset 28273216, docid=4469609979696008469, rowid=62106 of 241500 FAILED, Unknown blob row type: 83 at offset 28273458, docid=4469609979696008470, rowid=62107 of 241500 FAILED, Unknown blob row type: 63 at offset 28273696, docid=4469609979696008471, rowid=62108 of 241500 FAILED, Unknown blob row type: 100 at offset 28273937, docid=4469609979696008472, rowid=62109 of 241500 FAILED, Unknown blob row type: 83 at offset 28274338, docid=4469609979696008473, rowid=62110 of 241500 FAILED, Unknown blob row type: 7 at offset 28275990, docid=4469609979696008476, rowid=62113 of 241500 FAILED, Unknown blob row type: 62 at offset 28276485, docid=4469609979696008478, rowid=62115 of 241500 FAILED, Unknown blob row type: 144 at offset 28276721, docid=4469609979696008479, rowid=62116 of 241500 FAILED, Unknown blob row type: 116 at offset 28276963, docid=4469609979696008480, rowid=62117 of 241500 FAILED, Unknown blob row type: 7 at offset 28277430, docid=4469609979696008481, rowid=62118 of 241500 FAILED, Unknown blob row type: 116 at offset 28278515, docid=4469609979696008483, rowid=62120 of 241500 FAILED, Unknown blob row type: 116 at offset 28279011, docid=4469609979696008484, rowid=62121 of 241500 FAILED, Unknown blob row type: 240 at offset 28279704, docid=4469609979696008485, rowid=62122 of 241500 FAILED, Unknown blob row type: 62 at offset 28280421, docid=4469609979696008487, rowid=62124 of 241500 FAILED, Unknown blob row type: 7 at offset 28281878, docid=4469609979696008490, rowid=62127 of 241500 FAILED, Unknown blob row type: 62 at offset 28282375, docid=4469609979696008491, rowid=62128 of 241500 FAILED, Unknown blob row type: 152 at offset 28284660, docid=4469609979696008495, rowid=62132 of 241500 FAILED, Unknown blob row type: 62 at offset 28285141, docid=4469609979696008496, rowid=62133 of 241500 FAILED, Unknown blob row type: 62 at offset 28285637, docid=4469609979696008497, rowid=62134 of 241500 FAILED, Unknown blob row type: 10 at offset 28286330, docid=4469609979696008498, rowid=62135 of 241500 FAILED, Unknown blob row type: 131 at offset 28286568, docid=4469609979696008499, rowid=62136 of 241500 FAILED, Unknown blob row type: 143 at offset 28286809, docid=4469609979696008500, rowid=62137 of 241500 FAILED, Unknown blob row type: 7 at offset 28287206, docid=4469609979696008501, rowid=62138 of 241500 FAILED, Unknown blob row type: 63 at offset 28287921, docid=4469609979696008503, rowid=62140 of 241500 FAILED, Unknown blob row type: 116 at offset 28288163, docid=4469609979696008504, rowid=62141 of 241500 FAILED, Unknown blob row type: 74 at offset 28288641, docid=4469609979696008506, rowid=62143 of 241500 FAILED, Unknown blob row type: 116 at offset 28288899, docid=4469609979696008507, rowid=62144 of 241500 FAILED, Unknown blob row type: 127 at offset 28289392, docid=4469609979696008508, rowid=62145 of 241500 FAILED, Unknown blob row type: 120 at offset 28289801, docid=4469609979696008509, rowid=62146 of 241500 FAILED, Unknown blob row type: 7 at offset 28291270, docid=4469609979696008512, rowid=62149 of 241500 FAILED, Unknown blob row type: 252 at offset 28291800, docid=4469609979696008514, rowid=62151 of 241500 FAILED, Unknown blob row type: 160 at offset 28293272, docid=4469609979696008518, rowid=62155 of 241500 FAILED, Unknown blob row type: 12 at offset 28293802, docid=4469609979696008519, rowid=62156 of 241500 FAILED, Unknown blob row type: 191 at offset 28294592, docid=4469609979696008521, rowid=62158 of 241500 FAILED, Unknown blob row type: 23 at offset 28295129, docid=4469609979696008522, rowid=62159 of 241500 FAILED, Unknown blob row type: 116 at offset 28295667, docid=4469609979696008523, rowid=62160 of 241500 FAILED, Unknown blob row type: 116 at offset 28296195, docid=4469609979696008524, rowid=62161 of 241500 FAILED, Unknown blob row type: 127 at offset 28296944, docid=4469609979696008527, rowid=62164 of 241500 FAILED, Unknown blob row type: 127 at offset 28297648, docid=4469609979696008528, rowid=62165 of 241500 FAILED, Unknown blob row type: 123 at offset 28298984, docid=4469609979696008532, rowid=62169 of 241500 FAILED, Unknown blob row type: 152 at offset 28302340, docid=4469609979696008536, rowid=62173 of 241500 FAILED, Unknown blob row type: 15 at offset 28304314, docid=4469609979696008537, rowid=62174 of 241500 FAILED, Unknown blob row type: 62 at offset 28305079, docid=4469609979696008539, rowid=62176 of 241500 FAILED, Unknown blob row type: 152 at offset 28305316, docid=4469609979696008540, rowid=62177 of 241500 FAILED, Unknown blob row type: 85 at offset 28305554, docid=4469609979696008541, rowid=62178 of 241500 FAILED, Unknown blob row type: 116 at offset 28305795, docid=4469609979696008542, rowid=62179 of 241500 FAILED, Unknown blob row type: 16 at offset 28306458, docid=4469609979696008544, rowid=62181 of 241500 FAILED, Unknown blob row type: 116 at offset 28306931, docid=4469609979696008545, rowid=62182 of 241500 FAILED, Unknown blob row type: 7 at offset 28307430, docid=4469609979696008547, rowid=62184 of 241500 FAILED, Unknown blob row type: 249 at offset 28307897, docid=4469609979696008548, rowid=62185 of 241500 FAILED, Unknown blob row type: 62 at offset 28308133, docid=4469609979696008549, rowid=62186 of 241500 FAILED, Unknown blob row type: 175 at offset 28311473, docid=4469609979696008553, rowid=62190 of 241500 FAILED, Unknown blob row type: 255 at offset 28311952, docid=4469609979696008555, rowid=62192 of 241500 FAILED, Unknown blob row type: 7 at offset 28312822, docid=4469609979696008557, rowid=62194 of 241500 FAILED, Unknown blob row type: 208 at offset 28313585, docid=4469609979696008559, rowid=62196 of 241500 FAILED, Unknown blob row type: 219 at offset 28313832, docid=4469609979696008560, rowid=62197 of 241500 FAILED, Unknown blob row type: 238 at offset 28314329, docid=4469609979696008561, rowid=62198 of 241500 FAILED, Unknown blob row type: 152 at offset 28314852, docid=4469609979696008562, rowid=62199 of 241500 FAILED, Unknown blob row type: 7 at offset 28315318, docid=4469609979696008563, rowid=62200 of 241500 FAILED, Unknown blob row type: 160 at offset 28315555, docid=4469609979696008564, rowid=62201 of 241500 FAILED, Unknown blob row type: 160 at offset 28316084, docid=4469609979696008565, rowid=62202 of 241500 FAILED, Unknown blob row type: 195 at offset 28317175, docid=4469609979696008566, rowid=62203 of 241500 FAILED, Unknown blob row type: 117 at offset 28317663, docid=4469609979696008568, rowid=62205 of 241500 FAILED, Unknown blob row type: 253 at offset 28318061, docid=4469609979696008569, rowid=62206 of 241500 FAILED, Unknown blob row type: 230 at offset 28318302, docid=4469609979696008570, rowid=62207 of 241500 FAILED, Unknown blob row type: 65 at offset 28318699, docid=4469609979696008571, rowid=62208 of 241500 FAILED, Unknown blob row type: 43 at offset 28320300, docid=4469609979696008575, rowid=62212 of 241500 checking attribute blocks index... checking kill-list... checking dead row map... checking doc-id lookup... check FAILED, 99 of 164193 failures reported, 11.5 sec elapsed checking disk chunk, extension 3, 3(4)... checking schema... checking dictionary... checking data... checking rows... checking attribute blocks index... checking kill-list... checking dead row map... checking doc-id lookup... check FAILED, 99 of 164193 failures reported, 12.2 sec elbut still the elapsed check FAILED, 99 of 164193 failures reported, 12.2 sec elapsed

I removed the index but still error remains same

tomatolog commented 1 year ago

indextool said you have bad index - you need truncate your index and repopulate it from scratch

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Feel free to re-open the issue in case it becomes actual.