Closed Philippe23 closed 2 weeks ago
Philippe23
commented
4 months ago Unfortunately, I didn't have debug symbols being installed, so I just have the callstacks. I think I've installed debug symbols for the program at this point, so if/when it happens again, I may have some more info.
Philippe23
commented
4 months ago Callstack:
Core was generated by `/usr/libexec/dovecot/indexer-worker'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ffb1aa8735c in ?? () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7ffb17e006c0 (LWP 12050))]
(gdb) bt
#0 0x00007ffb1aa8735c in ?? () from /lib64/libc.so.6
#1 0x00007ffb1aa371f6 in raise () from /lib64/libc.so.6
#2 0x00007ffb1aa1f8f7 in abort () from /lib64/libc.so.6
#3 0x00007ffb1a243f22 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#4 0x00007ffb1a25697c in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#5 0x00007ffb1a2569e7 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#6 0x00007ffb1a256c47 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#7 0x00007ffb1a2439cc in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#8 0x00007ffb180ff985 in ?? () from /usr/lib64/libxapian.so.30
#9 0x00007ffb180fddf8 in Xapian::Document::Internal::add_term(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libxapian.so.30
#10 0x00007ffb18318f75 in XDoc::create_document (this=this@entry=0x555c4fdf9010, verbose=0,
title=0x555c4fc2fb60 "DW #267 (Holding.Order Information,/home/USER1/.maildir/xapian-indexes/db_23392717807b795b6d200000d09efc50) - ") at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/basic_string.tcc:242
#11 0x00007ffb1832cdc9 in XDocsWriter::worker (this=0x555c4fc3d130)
at /usr/src/debug/net-mail/dovecot-fts-xapian-1.7.14/fts-xapian-1.7.14/src/fts-backend-xapian-functions.cpp:843
#12 0x00007ffb1a2876a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#13 0x00007ffb1aa85652 in ?? () from /lib64/libc.so.6
#14 0x00007ffb1ab018ac in ?? () from /lib64/libc.so.6#11 0x00007ffb1832cdc9 in XDocsWriter::worker (this=0x555c4fc3d130)
at /usr/src/debug/net-mail/dovecot-fts-xapian-1.7.14/fts-xapian-1.7.14/src/fts-backend-xapian-functions.cpp:843
843 doc->create_document(verbose,title);
(gdb) info locals
start_time = 1722164636509
n = 3
err = false
i = 2
newdoc = 0
doc = 0x555c4fdf9010
(gdb) print *this
$9 = {dbpath = 0x555c4fbd5a50 "/home/USER1/.maildir/xapian-indexes/db_23392717807b795b6d200000d09efc50",
docs = 0x555c4fc1ed00, m = 0x555c4fbd0690, terminated = false, dbw = 0x555c4fb43280, verbose = 0,
batch = 0x555c4fbd06c8, totaldocs = 0x555c4fbd06c0, t = 0x555c4fc1de40,
title = 0x555c4fc2fb60 "DW #267 (Holding.Order Information,/home/USER1/.maildir/xapian-indexes/db_23392717807b795b6d200000d09efc50) - ", pos = 11}
(gdb) print *doc
$1 = {data = 0x7ffb17382010, strings = 0x555c4fdce990, headers = 0x555c4fb2a250, uid = 11964, size = 15,
stems = 50001, tsize = 9878, uterm = 0x555c4fbc1bb0 "Q11964", xdoc = 0x7ffadae9c7b0}strings & headers are both empty std::vectors.
(gdb) list
838 if(verbose>0) syslog(LOG_INFO,"%sProcessing #%ld (%ld/%ld) %s",title,doc->uid,i+1,n,doc->getSummary().c_str());
839 pos=10;
840 doc->populate_stems(verbose,title);
841 if(verbose>0) syslog(LOG_INFO,"%sCreating doc #%ld (%ld/%ld) %s",title,doc->uid,i+1,n,doc->getSummary().c_str());
842 pos=11;
843 doc->create_document(verbose,title);
844 if(verbose>0) syslog(LOG_INFO,"%sPushing Doc %ld (%ld/%ld) with %ld strings and %ld stems",title,doc->uid,i+1,n,doc->size,doc->stems);
845 if(doc->stems > 0)
846 {
847 pos=12;Logs:
Jul 28 06:48:27 myhostname dovecot[2176]: indexer-worker: Error: cannot allocate memory for thread-local data: ABORT
Jul 28 06:48:27 myhostname dovecot[2176]: indexer-worker(USER1)<8623><84I/VkwedsMmTRZ9:INRmDpohpmavIQAA0J78UA>: Fatal: master: service(indexer-worker): child 8623 returned error 127
Jul 28 06:48:30 myhostname dovecot[2176]: indexer-worker(USER1)<8886><84I/VkwedsMmTRZ9:aD3YLPshpma2IgAA0J78UA>: FTS Xapian: Waiting for all pending documents to be processed (Sleep5)
Jul 28 07:05:11 myhostname dovecot[2176]: indexer-worker(USER1)<11426><84I/VkwedsMmTRZ9:mFZODjolpmaiLAAA0J78UA>: Fatal: master: service(indexer-worker): child 11426 killed with signal 6 (core dumped)Footnote: There seem to be a few different PIDs. Not sure if that's expected, or if I'm grabbing a few issues, but only one's creating a core. (?)
I'm realizing I don't have Xapian's debug symbols either. Gonna' rebuild that as well with detached debug symbols.
I should point out that I've got 11.5 gigs of swap free.
And I'm linking against Xapian v1.4.25.
Philippe23
commented
4 months ago Core was generated by `/usr/libexec/dovecot/indexer-worker'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f3c0af2c35c in ?? () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f3c0785b6c0 (LWP 27298))]
(gdb) bt
#0 0x00007f3c0af2c35c in ?? () from /lib64/libc.so.6
#1 0x00007f3c0aedc1f6 in raise () from /lib64/libc.so.6
#2 0x00007f3c0aec48f7 in abort () from /lib64/libc.so.6
#3 0x00007f3c0a6e8f22 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#4 0x00007f3c0a6fb97c in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#5 0x00007f3c0a6fb9e7 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#6 0x00007f3c0a6fbc47 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#7 0x00007f3c0a6e89cc in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#8 0x00007f3c085a49a5 in std::__new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::allocate (this=<optimized out>, __n=1)
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/new_allocator.h:151
#9 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > > >::allocate (__n=1, __a=...)
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/alloc_traits.h:482
#10 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::_M_get_node (this=0x7f3bd7bd8880) at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_tree.h:563
#11 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::_M_create_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm> > (this=0x7f3bd7bd8880) at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_tree.h:613
#12 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::_Auto_node::_Auto_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm> > (__t=..., this=<synthetic pointer>)
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_tree.h:1637
#13 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::_M_emplace_hint_unique<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm> > (this=this@entry=0x7f3bd7bd8880, __pos=__pos@entry={...})
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_tree.h:2462
#14 0x00007f3c085a2e18 in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::emplace_hint<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm> > (__pos=..., this=0x7f3bd7bd8880)
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_map.h:640
#15 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::insert<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OmDocumentTerm> > (__x=..., this=0x7f3bd7bd8880)
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_map.h:860
#16 Xapian::Document::Internal::add_term (this=0x7f3bd7bd8810, tname=..., wdfinc=<optimized out>) at api/omdocument.cc:502
#17 0x00007f3c085a2f3b in Xapian::Document::add_term (this=<optimized out>, tname=..., wdfinc=<optimized out>) at api/omdocument.cc:146
#18 0x00007f3c087bdf75 in XDoc::create_document (this=this@entry=0x5650a507bec0, verbose=0, title=0x5650a500c330 "DW #253 (Holding.Order Information,/home/USER1/.maildir/xapian-indexes/db_23392717807b795b6d200000d09efc50) - ")
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/basic_string.tcc:242
#19 0x00007f3c087d1dc9 in XDocsWriter::worker (this=0x5650a51c1b30) at /usr/src/debug/net-mail/dovecot-fts-xapian-1.7.14/fts-xapian-1.7.14/src/fts-backend-xapian-functions.cpp:843
#20 0x00007f3c0a72c6a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#21 0x00007f3c0af2a652 in ?? () from /lib64/libc.so.6
#22 0x00007f3c0afa68ac in ?? () from /lib64/libc.so.6(gdb) up
#16 Xapian::Document::Internal::add_term (this=0x7f3bd7bd8810, tname=..., wdfinc=<optimized out>) at api/omdocument.cc:502
502 terms.insert(make_pair(tname, std::move(newterm)));
(gdb) list
497 map<string, OmDocumentTerm>::iterator i;
498 i = terms.find(tname);
499 if (i == terms.end()) {
500 ++termlist_size;
501 OmDocumentTerm newterm(wdfinc);
502 terms.insert(make_pair(tname, std::move(newterm)));
503 } else {
504 if (i->second.increase_wdf(wdfinc))
505 ++termlist_size;
506 }
(gdb) print terms
$4 = std::map with 40230 elements = {["Q11923"] **snip**#8 0x00007f3c085a49a5 in std::__new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OmDocumentTerm> > >::allocate (this=<optimized out>, __n=1)
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/new_allocator.h:151
151 return static_cast<_Tp*>(_GLIBCXX_OPERATOR_NEW(__n * sizeof(_Tp)));
(gdb) print __n
$12 = 1
(gdb) print sizeof(_Tp)
$13 = 96So it's got a good size document going (40k of terms), but the allocation isn't huge, just 96 bytes.
grosjo
commented
4 months ago Weird. For your last comment : 96 bytes for 40k elements ? Meaning most, if not all, are empty ?
Philippe23
commented
3 months ago 96 bytes for 40k elements ? Meaning most, if not all, are empty ?
No, the 96 bytes appears to be just a node in a data structure (red-black tree). So it's a straw-that-broke-the-camel's-back situation.
My config has vsz_limit = 1G, so I'm guessing that dovecot proper enforces this memory cap, and we're hitting that cap. (Also, this is why having all that free swap at a system-level doesn't matter.)
Since I don't have any 1G messages, perhaps there's a memory leak?
I'll also try raising vsz_limit a little and see if it keeps happening.
But feel free to suggest other things to try to help track down what's going on.
Also, I'm not sure if there's a way for you to catch the allocation exception and recover from it.
grosjo
commented
3 months ago 1G for 40k terms, this seems to much, as each term can not be more than than 245 chars
grosjo
commented
3 months ago I found a memory leaj. Can you try latest git ?
Philippe23
commented
3 months ago Upgraded to latest, and returned to the 1G limit I had before. I'll keep an eye on it for a few days and let you know if I see it again or not.
Philippe23
commented
3 months ago Not sure if it's related or a new crash, but I just got this twice:
Logs:
Aug 6 15:58:18 myhostname xapian-docswriter[23673]: DW #264 (Archived Trash.2024,/home/USER1/.maildir/xapian-indexes/db_c3bf2b05772d8066180f0000d09efc50) - Can't add document2 : std::exception
Aug 6 15:58:18 myhostname xapian-docswriter[23673]: DW #264 (Archived Trash.2024,/home/USER1/.maildir/xapian-indexes/db_c3bf2b05772d8066180f0000d09efc50) - Retrying (/home/USER1/.maildir/xapian-indexes/db_c3bf2b05772d8066180f0000d09efc50) from std::exception
Aug 6 15:58:18 myhostname kernel: indexer-worker[23957]: segfault at 9 ip 00007fb8036b3d28 sp 00007fb8028f8ae0 error 4 in libxapian.so.30.12.6[7fb8035a4000+1ba000] likely on CPU 1 (core 0, socket 0)
Aug 6 15:58:18 myhostname kernel: Code: 5c e9 bc 12 ef ff 0f 1f 40 00 5b 5d 41 5c c3 90 66 90 f3 0f 1e fa 41 57 49 89 ff 41 56 41 55 41 54 49 89 f4 55 53 48 83 ec 18 <0f> b7 6f 09 48 89 4c 24 08 66 c1 c5 08 0f b7 ed 83 fa ff 0f 84 ff
Aug 6 15:58:59 myhostname dovecot[10243]: indexer-worker(USER1)<23673><yjsoLwkfA8EmTRZ9:a6d9Hw2AsmZ5XAAA0J78UA>: Fatal: master: service(indexer-worker): child 23673 killed with signal 11 (core dumped)Backtrace:
Core was generated by `/usr/libexec/dovecot/indexer-worker'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fb8036b3d28 in do_bswap (value=<optimized out>) at ./common/wordaccess.h:47
47 return __builtin_bswap16(value);
[Current thread is 1 (Thread 0x7fb8028f96c0 (LWP 23957))]
(gdb) bt
#0 0x00007fb8036b3d28 in do_bswap (value=<optimized out>) at ./common/wordaccess.h:47
#1 do_unaligned_read<unsigned short> (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:122
#2 unaligned_read2 (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:165
#3 Glass::DIR_END (b=0x0) at backends/glass/glass_table.h:114
#4 GlassTable::find_in_leaf (p=0x0, item=..., c=11, exact=exact@entry=@0x7fb8028f8b37: false) at backends/glass/glass_table.cc:428
#5 0x00007fb8036b5d24 in GlassTable::find (this=this@entry=0x5645fa45c530, C_=C_@entry=0x5645fa45c5f8) at backends/glass/glass_table.h:175
#6 0x00007fb8036b5d85 in GlassTable::key_exists (this=this@entry=0x5645fa45c530, key=<incomplete sequence \340>) at backends/glass/glass_table.cc:1505
#7 0x00007fb8036ab274 in GlassPostListTable::merge_doclen_changes (this=this@entry=0x5645fa45c530, doclens=std::map with 766 elements = {...}) at backends/glass/glass_postlist.cc:1149
#8 0x00007fb8036a2470 in Inverter::flush_doclengths (table=..., this=0x5645fa45d130) at backends/glass/glass_inverter.cc:158
#9 Inverter::flush (this=this@entry=0x5645fa45d130, table=...) at backends/glass/glass_inverter.cc:218
#10 0x00007fb803692f9f in GlassWritableDatabase::flush_postlist_changes (this=0x5645fa45c160) at backends/glass/glass_database.cc:1064
#11 0x00007fb803693ba5 in GlassWritableDatabase::commit (this=0x5645fa45c160) at backends/glass/glass_database.cc:1043
#12 0x00007fb8035ebf81 in Xapian::WritableDatabase::commit (this=0x5645fa3e00c0) at api/omdatabase.cc:864
#13 0x00007fb80382005a in XDocsWriter::worker (this=0x5645fa513450) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:908
#14 0x00007fb80577a6a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#15 0x00007fb805f78652 in ?? () from /lib64/libc.so.6
#16 0x00007fb805ff48ac in ?? () from /lib64/libc.so.6Code for frame #13 + data I can find from the locals:
(gdb) list
903 {
904 pos=16;
905 syslog(LOG_WARNING,"%s Retrying (%s) from %s",title,dbpath,err_s.c_str());
906 try
907 {
908 dbw->commit();
909 pos=17;
910 dbw->replace_document(doc->uterm,*(doc->xdoc));
911 (*totaldocs)++;
912 }
913 catch(Xapian::Error e)
914 {
915 syslog(LOG_ERR,"%sCan't add document3 : %s - %s %s",title,e.get_type(),e.get_msg().c_str(),e.get_error_string());
916 }
917 catch(std::exception e)
918 {
919 syslog(LOG_ERR,"%sCan't add document4 : %s",title,e.what());
920 }
921 pos=18;
922 }
(gdb) info locals
lck = {_M_device = 0x5645fa40f400, _M_owns = true}
err_s = "std::exception"
start_time = 1722974297079
n = 3
err = <optimized out>
i = <optimized out>
newdoc = 2
doc = 0x5645fa5a80f0
(gdb) print *doc
$1 = {data = 0x0, strings = 0x5645fa5e1900, headers = 0x5645fa46b910, uid = 769, size = 17, stems = 28517, tsize = 9819, uterm = 0x5645fa54fdb0 "Q769", xdoc = 0x5645fa51d520}
gdb) print *doc->strings
$2 = std::vector of length 0, capacity 32
(gdb) print *doc->headers
$3 = std::vector of length 0, capacity 32Let me know if you want me to open a different issue for this one.
Oh, and if it's relevant, I'm linked against Xapian 1.4.25.
grosjo
commented
3 months ago This seems to be either another memory leak, or a low memory on your server.
How much memory yo allocated to the indexer ?
grosjo
commented
3 months ago Can you provide the few lines before "Can't add document2" ?
grosjo
commented
3 months ago please also use latest git
Philippe23
commented
3 months ago Can you provide the few lines before "Can't add document2" ?
867 #pragma GCC diagnostic push
868 #pragma GCC diagnostic ignored "-Wdeprecated-declarations"
869 while(!(lck.try_lock_for(std::chrono::milliseconds(1000 + std::rand() % 1000))))
870 {
871 if(verbose>1) syslog(LOG_INFO,"%sMutex : Waiting unlock... (thread)",title);
872 }
873 #pragma GCC diagnostic pop
874 if(verbose>1) syslog(LOG_INFO,"%sMutex : Lock acquired (thread)",title);
875 std::string err_s; err_s.clear();
876 try
877 {
878 pos=13;
879 dbw->replace_document(doc->uterm,*(doc->xdoc));
880 (*totaldocs)++;
881 (*batch)++;
882 if((*batch) > XAPIAN_WRITING_CACHE)
883 {
884 syslog(LOG_INFO,"%s Committing %ld docs (vs %ld limit)",title,(*batch),XAPIAN_WRITING_CACHE );
885 pos=15;
886 dbw->commit();
887 *batch = 0;
888 }
889 }
890 catch(Xapian::Error e)
891 { 892 syslog(LOG_WARNING,"%sCan't add document1 : %s - %s %s",title,e.get_type(), e.get_msg().c_str(),e.get_error_string());
893 err=true;
894 err_s.append(e.get_type());
895 }
896 catch(std::exception e)
897 {
898 syslog(LOG_WARNING,"%sCan't add document2 : %s",title,e.what());
899 err=true;
900 err_s.append(e.what());
901 }
902 if(err)
903 {
904 pos=16;
905 syslog(LOG_WARNING,"%s Retrying (%s) from %s",title,dbpath,err_s.c_str());
906 try
907 {
908 dbw->commit();
909 pos=17;
910 dbw->replace_document(doc->uterm,*(doc->xdoc));
911 (*totaldocs)++;
912 }
913 catch(Xapian::Error e)
914 { 915 syslog(LOG_ERR,"%sCan't add document3 : %s - %s %s",title,e.get_type(), e.get_msg().c_str(),e.get_error_string());
916 }
917 catch(std::exception e)
918 {
919 syslog(LOG_ERR,"%sCan't add document4 : %s",title,e.what());
920 }
921 pos=18;
922 }
923 }
924 pos=19;
925 delete(doc);
926 newdoc++;
927 } 928 if(verbose>0) syslog(LOG_INFO,"%sWrote %ld new docs in %ld ms",title,newdoc,fts_backend_xapian_curr ent_time() - start_time);The additional context you requested before I upgrade to latest.
grosjo
commented
3 months ago I meant, can you redo your test with the latest git ?
grosjo
commented
3 months ago Any crash ?
Philippe23
commented
3 months ago I haven't had a crash since August 6th. I've been running the latest since August 13th. (August 13th is also the last time I've restarted my dovecot process.)
Philippe23
commented
3 months ago Core was generated by `/usr/libexec/dovecot/indexer-worker'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007efe631dcd28 in do_bswap (value=<optimized out>)
at ./common/wordaccess.h:47
47 return __builtin_bswap16(value);
[Current thread is 1 (Thread 0x7efe617116c0 (LWP 5949))]
(gdb) bt
#0 0x00007efe631dcd28 in do_bswap (value=<optimized out>) at ./common/wordaccess.h:47
#1 do_unaligned_read<unsigned short> (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:122
#2 unaligned_read2 (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:165
#3 Glass::DIR_END (b=0x0) at backends/glass/glass_table.h:114
#4 GlassTable::find_in_leaf (p=0x0, item=..., c=-1, exact=exact@entry=@0x7efe61710a87: false) at backends/glass/glass_table.cc:428
#5 0x00007efe631ded24 in GlassTable::find (this=0x7efe54046090, C_=0x7efe28bac598) at backends/glass/glass_table.h:175
#6 0x00007efe631b7488 in GlassCursor::find_entry (this=0x7efe2908e230, key="Q4238") at backends/glass/glass_cursor.cc:164
#7 0x00007efe631d0c1d in GlassPostList::init (this=this@entry=0x7efe5c1777a0) at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/unique_ptr.h:199
#8 0x00007efe631d1064 in GlassPostList::GlassPostList (this=this@entry=0x7efe5c1777a0, this_db_=..., term_="Q4238", keep_reference=keep_reference@entry=true) at backends/glass/glass_postlist.cc:708
#9 0x00007efe631ba827 in GlassWritableDatabase::open_post_list (this=0x7efe54045cc0, tname="Q4238") at ./include/xapian/intrusive_ptr.h:155
#10 0x00007efe6314f7ad in Xapian::Database::Internal::replace_document (this=0x7efe54045cc0, unique_term=..., document=...) at backends/database.cc:197
#11 0x00007efe63348f6a in XDocsWriter::worker (this=0x7efe5c0f0340) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:1025
#12 0x00007efe652a36a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#13 0x00007efe65aa1652 in ?? () from /lib64/libc.so.6
#14 0x00007efe65b1d8ac in ?? () from /lib64/libc.so.6Aug 24 07:31:55 myhostname dovecot[4322]: indexer-worker(USER7)<5932><Er+FJaXEyWYoFwAA0J78UA:6JAaNaXEyWYsFwAA0J78UA>: FTS Xapian: Waiting for all pending documents (1) to be processed (Sleep5) with 2 threads
Aug 24 07:34:12 myhostname xapian-docswriter[5932]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Can't write doc2 : std::exception
Aug 24 07:34:12 myhostname xapian-docswriter[5932]: FTS Xapian: Can not allocate ICU translator (2)
Aug 24 07:34:12 myhostname kernel: indexer-worker[5949]: segfault at 9 ip 00007efe631dcd28 sp 00007efe61710a30 error 4 in libxapian.so.30.12.6[7efe630cd000+1ba000] likely on CPU 1 (core 0, socket 0)
Aug 24 07:34:12 myhostname kernel: Code: 5c e9 bc 12 ef ff 0f 1f 40 00 5b 5d 41 5c c3 90 66 90 f3 0f 1e fa 41 57 49 89 ff 41 56 41 55 41 54 49 89 f4 55 53 48 83 ec 18 <0f> b7 6f 09 48 89 4c 24 08 66 c1 c5 08 0f b7 ed 83 fa ff 0f 84 ff
Aug 24 07:34:51 myhostname dovecot[4322]: indexer-worker(USER1)<5932><6Lv6JGwgWs8mTRZ9:0D63G7vEyWYsFwAA0J78UA>: Fatal: master: service(indexer-worker): child 5932 killed with signal 11 (core dumped)I suspect this one is an upstream bug.
grosjo
commented
2 months ago This looks like a memory leak
Philippe23
commented
2 months ago Could be -- certainly happened when memory was low, but it doesn't look like you're directly causing the crash from what I see. All the pointers you're passing in look like they're non-NULL from what I see. So my gut says that it's upstream not handling an allocation failure properly. I guess, "why not both?" could apply here.
grosjo
commented
2 months ago Kindly set verbose=1 and use latest git
Philippe23
commented
2 months ago Latest has a ton of warnings like this:
fts-backend-xapian-functions.cpp:951:83: warning: format not a string literal and no format arguments [-Wformat-security]
951 | if(verbose>0) { s=title; s.append("No-op"); syslog(LOG_INFO,s.c_str()); }
| ~~~~~~^~~~~~~~~~~~~~~~~~~~You should do these like this 👇 if you're just trying to output a code-built string:
syslog(LOG_INFO, "%s", s.c_str());
// ^^^^Otherwise, you could have a % format character in the string you're trying to output and the library will assume there's an arg passed that contains what it should read (but there isn't).
With the "%s" format string you're saying, output this string arg as-is.
Philippe23
commented
2 months ago Made a PR to fix 'em for you: https://github.com/grosjo/fts-xapian/pull/171
grosjo
commented
2 months ago Approved your PR.
Let's see now the logs before the crash
Philippe23
commented
2 months ago Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4235 #lines=6 #stems=0 #mem=2.217773 KB status=1
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=0 #mem=2.217773 KB status=1 : Populate 6 headers with strings
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=0 #mem=0.000000 KB status=1 : Populate 6 / 6 Header=XBDY TextLength=2067
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=2723 #mem=32.796875 KB status=1 : Populate 5 / 6 Header=XCT TextLength=30
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=2895 #mem=34.718750 KB status=1 : Populate 4 / 6 Header=S TextLength=10
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=2905 #mem=34.777344 KB status=1 : Populate 3 / 6 Header=XTO TextLength=54
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=3293 #mem=40.223633 KB status=1 : Populate 2 / 6 Header=A TextLength=46
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=3604 #mem=43.476562 KB status=1 : Populate 1 / 6 Header=XMID TextLength=36
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4235 #lines=6 #stems=3606 #mem=43.554688 KB status=1 : Done populating in 61 ms (59114 stems/sec)
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: STEMS populated : 3606 [ A_co A_com A_com_ A_pa A_par A_para A_paral A_parall A_paralla A_parallax XTOuil_gmail_com_ XTOuil_gmail_com XTOuil_gmail_co XTOuil_gmail_c XTOuil_gmail_ XTOuil_gmail XTOuil_gmai XTOuil_gma XTOuil_gm
]
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Creating Xapian doc : Doc 4235 #lines=6 #stems=3606 #mem=43.554688 KB status=2
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 3606 terms to doc (Doc 4235 #lines=6 #stems=3606 #mem=43.554688 KB status=2)
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4235 #lines=6 #stems=3606 #mem=43.554688 KB status=2)
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4235 #lines=6 #stems=3606 #mem=43.554688 KB status=3
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 139550 KB // Free = 471892 KB
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4235 #lines=6 #stems=3606 #mem=43.554688 KB status=3 Free memory : 460MB
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4236 #lines=7 #stems=0 #mem=0.791992 KB status=1
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=0 #mem=0.791992 KB status=1 : Populate 7 headers with strings
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=0 #mem=0.000000 KB status=1 : Populate 7 / 7 Header=XBDY TextLength=580
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=809 #mem=9.489258 KB status=1 : Populate 6 / 7 Header=XCT TextLength=30
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=981 #mem=11.411133 KB status=1 : Populate 5 / 7 Header=S TextLength=9
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=988 #mem=11.449219 KB status=1 : Populate 4 / 7 Header=XCC TextLength=42
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=1204 #mem=13.989258 KB status=1 : Populate 3 / 7 Header=XTO TextLength=35
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=1372 #mem=15.802734 KB status=1 : Populate 2 / 7 Header=A TextLength=46
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=1683 #mem=19.055664 KB status=1 : Populate 1 / 7 Header=XMID TextLength=36
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4236 #lines=7 #stems=1685 #mem=19.133789 KB status=1 : Done populating in 13 ms (129615 stems/sec)
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: STEMS populated : 1685 [ A_co A_com A_com_ A_pa A_par A_para A_paral A_parall A_paralla A_parallax XTOvela_com_ XTOvela_com XTOvela_co XTOvela_c XTOvela_ XTOvela XTOvel XTOuil XTOtreuil]
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Creating Xapian doc : Doc 4236 #lines=7 #stems=1685 #mem=19.133789 KB status=2
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 1685 terms to doc (Doc 4236 #lines=7 #stems=1685 #mem=19.133789 KB status=2)
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4236 #lines=7 #stems=1685 #mem=19.133789 KB status=2)
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4236 #lines=7 #stems=1685 #mem=19.133789 KB status=3
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 139569 KB // Free = 471892 KB
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4236 #lines=7 #stems=1685 #mem=19.133789 KB status=3 Free memory : 460MB
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:25:02 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - No-op
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - No-op
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 14 / 30 Header=XCT TextLength=26
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 13 / 30 Header=XCT TextLength=74
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 12 / 30 Header=XMID TextLength=37
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 11 / 30 Header=S TextLength=67
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 10 / 30 Header=XTO TextLength=50
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 9 / 30 Header=A TextLength=42
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 8 / 30 Header=XCT TextLength=93
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 7 / 30 Header=XBDY TextLength=562
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 6 / 30 Header=XCT TextLength=25
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 5 / 30 Header=XCT TextLength=63
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 4 / 30 Header=S TextLength=72
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 3 / 30 Header=XTO TextLength=74
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 2 / 30 Header=A TextLength=46
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Populate 1 / 30 Header=XMID TextLength=36
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=1 : Done populating in 3187 ms (15689 stems/sec)
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: STEMS populated : 50001 [ XBDY$12 XBDY$12_ XBDY$12_0 XBDY$12_00 XBDY//r XBDY//r2 XBDY//r20 XBDY//r20_ XBDY//r20_r XBDY//r20_rs XCTxt/html XCTxt/htm XCTxt/ht XCTxt/h XCTxt/ XCTutf_8 XCTutf_ XCTutf XCTtml]
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Creating Xapian doc : Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=2
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 50001 terms to doc (Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=2)
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - No-op
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=2)
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=3
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 140320 KB // Free = 470860 KB
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=3 Free memory : 459MB
Sep 17 06:25:03 myhostname xapian-docswriter[4867]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:25:05 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Can't write doc2 : std::exception
Sep 17 06:25:05 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=3
Sep 17 06:25:05 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 140320 KB // Free = 473644 KB
Sep 17 06:25:05 myhostname xapian-docswriter[4867]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4186 #lines=30 #stems=50001 #mem=778.699219 KB status=3 Free memory : 462MB
Sep 17 06:25:05 myhostname kernel: indexer-worker[4868]: segfault at 9 ip 00007f9cc0473d28 sp 00007f9cbf5b4930 error 4 in libxapian.so.30.12.6[7f9cc0364000+1ba000] likely on CPU 1 (core 0, socket 0)
Sep 17 06:25:05 myhostname kernel: Code: 5c e9 bc 12 ef ff 0f 1f 40 00 5b 5d 41 5c c3 90 66 90 f3 0f 1e fa 41 57 49 89 ff 41 56 41 55 41 54 49 89 f4 55 53 48 83 ec 18 <0f> b7 6f 09 48 89 4c 24 08 66 c1 c5 08 0f b7 ed 83 fa ff 0f 84 ff
Sep 17 06:25:53 myhostname dovecot[30347]: indexer-worker(USER1)<4867><kLoeD04itLwmTRZ9:d141OkRY6WYDEwAA0J78UA>: Fatal: master: service(indexer-worker): child 4867 killed with signal 11 (core dumped)Callstack:
#0 0x00007f9cc0473d28 in do_bswap (value=<optimized out>) at ./common/wordaccess.h:47
#1 do_unaligned_read<unsigned short> (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:122
#2 unaligned_read2 (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:165
#3 Glass::DIR_END (b=0x0) at backends/glass/glass_table.h:114
#4 GlassTable::find_in_leaf (p=0x0, item=..., c=11, exact=exact@entry=@0x7f9cbf5b4987: false) at backends/glass/glass_table.cc:428
#5 0x00007f9cc0475d24 in GlassTable::find (this=this@entry=0x7f9cb00158e0, C_=C_@entry=0x7f9cb00159a8) at backends/glass/glass_table.h:175
#6 0x00007f9cc04793f1 in GlassTable::get_exact_entry (this=this@entry=0x7f9cb00158e0, key="Q4186", tag="") at backends/glass/glass_table.cc:1489
#7 0x00007f9cc046b7cf in GlassPostListTable::merge_changes (this=0x7f9cb00158e0, term=..., changes=...) at backends/glass/glass_postlist.cc:1213
#8 0x00007f9cc0462353 in Inverter::flush_post_list (this=this@entry=0x7f9cb00164e0, table=..., term="Q4186") at backends/glass/glass_inverter.cc:170
#9 0x00007f9cc04517f9 in GlassWritableDatabase::open_post_list (this=0x7f9cb0015510, tname="Q4186") at backends/glass/glass_database.cc:1502
#10 0x00007f9cc03e67ad in Xapian::Database::Internal::replace_document (this=0x7f9cb0015510, unique_term=..., document=...) at backends/database.cc:197
#11 0x00007f9cc05dffb8 in XDocsWriter::worker (this=0x55a30f56d8d0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:1026
#12 0x00007f9cc253b6a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#13 0x00007f9cc2d39652 in ?? () from /lib64/libc.so.6
#14 0x00007f9cc2db58ac in ?? () from /lib64/libc.so.6Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 1986 terms to doc (Doc 4336 #lines=6 #stems=1986 #mem=21.737305 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4336 #lines=6 #stems=1986 #mem=21.737305 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4336 #lines=6 #stems=1986 #mem=21.737305 KB status=3
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4337 #lines=6 #stems=0 #mem=0.878906 KB status=1
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=0 #mem=0.878906 KB status=1 : Populate 6 headers with strings
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=0 #mem=0.000000 KB status=1 : Populate 6 / 6 Header=XBDY TextLength=700
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 143056 KB // Free = 538784 KB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4336 #lines=6 #stems=1986 #mem=21.737305 KB status=3 Free memory : 526MB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=747 #mem=8.060547 KB status=1 : Populate 5 / 6 Header=XCT TextLength=30
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=919 #mem=9.982422 KB status=1 : Populate 4 / 6 Header=S TextLength=18
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=997 #mem=10.642578 KB status=1 : Populate 3 / 6 Header=XTO TextLength=42
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4338 #lines=6 #stems=0 #mem=0.779297 KB status=1
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=0 #mem=0.779297 KB status=1 : Populate 6 headers with strings
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=0 #mem=0.000000 KB status=1 : Populate 6 / 6 Header=XBDY TextLength=603
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=1234 #mem=13.360352 KB status=1 : Populate 2 / 6 Header=A TextLength=46
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=1545 #mem=16.613281 KB status=1 : Populate 1 / 6 Header=XMID TextLength=36
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4337 #lines=6 #stems=1547 #mem=16.691406 KB status=1 : Done populating in 35 ms (44200 stems/sec)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: STEMS populated : 1547 [ A_co A_com A_com_ A_pa A_par A_para A_paral A_parall A_paralla A_parallax XTOuil XTOtreuil XTOtreui XTOtreu XTOtre XTOreuil XTOreui XTOreu XTOrence]
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Creating Xapian doc : Doc 4337 #lines=6 #stems=1547 #mem=16.691406 KB status=2
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 1547 terms to doc (Doc 4337 #lines=6 #stems=1547 #mem=16.691406 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=662 #mem=7.237305 KB status=1 : Populate 5 / 6 Header=XCT TextLength=30
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=834 #mem=9.159180 KB status=1 : Populate 4 / 6 Header=S TextLength=13
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=845 #mem=9.222656 KB status=1 : Populate 3 / 6 Header=XTO TextLength=42
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=1061 #mem=11.762695 KB status=1 : Populate 2 / 6 Header=A TextLength=46
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=1372 #mem=15.015625 KB status=1 : Populate 1 / 6 Header=XMID TextLength=36
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4338 #lines=6 #stems=1374 #mem=15.093750 KB status=1 : Done populating in 21 ms (65428 stems/sec)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: STEMS populated : 1374 [ A_co A_com A_com_ A_pa A_par A_para A_paral A_parall A_paralla A_parallax XTOy13_com_ XTOy13_com XTOy13_co XTOy13_c XTOy13_ XTOy13 XTOuil XTOtreuil XTOtreui]
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Creating Xapian doc : Doc 4338 #lines=6 #stems=1374 #mem=15.093750 KB status=2
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 1374 terms to doc (Doc 4338 #lines=6 #stems=1374 #mem=15.093750 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4338 #lines=6 #stems=1374 #mem=15.093750 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4338 #lines=6 #stems=1374 #mem=15.093750 KB status=3
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 143087 KB // Free = 537480 KB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4338 #lines=6 #stems=1374 #mem=15.093750 KB status=3 Free memory : 524MB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4339 #lines=6 #stems=0 #mem=0.784180 KB status=1
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=0 #mem=0.784180 KB status=1 : Populate 6 headers with strings
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=0 #mem=0.000000 KB status=1 : Populate 6 / 6 Header=XBDY TextLength=605
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=668 #mem=7.333984 KB status=1 : Populate 5 / 6 Header=XCT TextLength=30
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=840 #mem=9.255859 KB status=1 : Populate 4 / 6 Header=S TextLength=17
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=851 #mem=9.319336 KB status=1 : Populate 3 / 6 Header=XTO TextLength=43
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=1067 #mem=11.859375 KB status=1 : Populate 2 / 6 Header=A TextLength=46
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=1378 #mem=15.112305 KB status=1 : Populate 1 / 6 Header=XMID TextLength=34
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4339 #lines=6 #stems=1380 #mem=15.186523 KB status=1 : Done populating in 13 ms (106153 stems/sec)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: STEMS populated : 1380 [ A_co A_com A_com_ A_pa A_par A_para A_paral A_parall A_paralla A_parallax XTOy13_com_ XTOy13_com XTOy13_co XTOy13_c XTOy13_ XTOy13 XTOuil XTOtreuil XTOtreui]
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Creating Xapian doc : Doc 4339 #lines=6 #stems=1380 #mem=15.186523 KB status=2
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - adding 1380 terms to doc (Doc 4339 #lines=6 #stems=1380 #mem=15.186523 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4339 #lines=6 #stems=1380 #mem=15.186523 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4339 #lines=6 #stems=1380 #mem=15.186523 KB status=3
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 143102 KB // Free = 537476 KB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4339 #lines=6 #stems=1380 #mem=15.186523 KB status=3 Free memory : 524MB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - create_doc done (Doc 4337 #lines=6 #stems=1547 #mem=16.691406 KB status=2)
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Pushing : Doc 4337 #lines=6 #stems=1547 #mem=16.691406 KB status=3
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4340 #lines=12 #stems=0 #mem=12.164062 KB status=1
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4340 #lines=12 #stems=0 #mem=12.164062 KB status=1 : Populate 12 headers with strings
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4340 #lines=12 #stems=0 #mem=0.000000 KB status=1 : Populate 12 / 12 Header=XBDY TextLength=582
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Mem used = 143114 KB // Free = 537476 KB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Replace doc : Doc 4337 #lines=6 #stems=1547 #mem=16.691406 KB status=3 Free memory : 524MB
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #1 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4340 #lines=12 #stems=684 #mem=7.533203 KB status=1 : Populate 11 / 12 Header=XBDY TextLength=5130
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc done
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Searching doc
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Populating stems : Doc 4341 #lines=11 #stems=0 #mem=11.893555 KB status=1
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4341 #lines=11 #stems=0 #mem=11.893555 KB status=1 : Populate 11 headers with strings
Sep 17 06:29:33 myhostname xapian-docswriter[5563]: DW #2 (Sent,/home/USER1/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - Doc 4341 #lines=11 #stems=0 #mem=0.000000 KB status=1 : Populate 11 / 11 Header=XBDY TextLength=4825
Sep 17 06:29:33 myhostname kernel: indexer-worker[5634]: segfault at 230 ip 00007fb48cd32981 sp 00007fb488ca35c8 error 4 in libc.so.6[7fb48cc02000+158000] likely on CPU 0 (core 0, socket 0)
Sep 17 06:29:33 myhostname kernel: Code: 48 01 d0 eb 1b 0f 1f 40 00 f3 0f 1e fa 48 39 d1 0f 82 b3 d5 fb ff 0f 1f 00 f3 0f 1e fa 48 89 f8 48 83 fa 20 0f 82 af 00 00 00 <c5> fe 6f 06 48 83 fa 40 0f 87 3e 01 00 00 c5 fe 6f 4c 16 e0 c5 fe
Sep 17 06:29:45 myhostname dovecot[30347]: indexer-worker(USER1)<5563><kLoeD04itLwmTRZ9:+Ei2CDJZ6Wa7FQAA0J78UA>: Fatal: master: service(indexer-worker): child 5563 killed with signal 11 (core dumped)Callstack:
#0 0x00007fb48cd32981 in ?? () from /lib64/libc.so.6
#1 0x00007fb48a4f7cab in XNGram::add_stem (this=this@entry=0x56528841dd30, d=d@entry=0x7fb488ca36f0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:597
#2 0x00007fb48a50da9a in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf878d0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:521
#3 0x00007fb48a50d974 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf5e320) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:496
#4 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bff2b10) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#5 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfd89e0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#6 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf373b0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#7 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bff1e20) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#8 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bff1380) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#9 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf7b530) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#10 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfe07b0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#11 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bff6710) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#12 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf880a0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#13 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfeb0e0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
... snip ...
#388 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf847f0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#389 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf29950) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#390 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfda560) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#391 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf92520) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#392 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf772d0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#393 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfda010) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#394 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfd9a70) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#395 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf454c0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#396 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45b21f810) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#397 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfe8110) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#398 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45b4f2c90) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#399 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf6f930) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#400 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf33770) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#401 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf9c010) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#402 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bf6fa00) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#403 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x7fb45bfebf90) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#404 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x5652885f99a0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#405 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x565288579150) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#406 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=d@entry=0x56528846a5e0) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#407 0x00007fb48a50d969 in XNGram::add (this=this@entry=0x56528841dd30, d=<optimized out>) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:495
#408 0x00007fb48a50df52 in XDoc::populate_stems (this=this@entry=0x5652886130a0, verbose=1, title=0x56528817e930 "DW #2 (Sent,/home/peep/.maildir/xapian-indexes/db_90751f2b357a795b611b0000d09efc50) - ")
at /usr/lib/gcc/x86_64-pc-linux-gnu/13/include/g++-v13/bits/stl_vector.h:1128
#409 0x00007fb48a50ec0d in XDocsWriter::worker (this=0x5652881a3900) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:957
#410 0x00007fb48c46a6a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#411 0x00007fb48cc68652 in ?? () from /lib64/libc.so.6
#412 0x00007fb48cce48ac in ?? () from /lib64/libc.so.6Naturally, there's a ton of log lines. Let me know if there's something I should search for near those core dumps.
grosjo
commented
1 month ago Hi @Philippe23 , kindly test latest git
symphorien
commented
1 month ago I am seeing the same issue on the 1.7.15 tag
server # 2024-10-06T20:43:31.801363+00:00 server dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::out_of_range'
server # [ 96.199554] dovecot[1292]: indexer-worker: Error: terminate called after throwing an instance of 'std::out_of_range'
server # [ 96.201236] dovecot[1292]: indexer-worker: Error: what(): vector::_M_range_check: __n (which is 18446744073709551615) >= this->size() (which is 1)
server # 2024-10-06T20:43:31.804317+00:00 server dovecot: indexer-worker: Error: what(): vector::_M_range_check: __n (which is 18446744073709551615) >= this->size() (which is 1)
server # [ 96.216891] systemd-coredump[1503]: Process 1498 (indexer-worker) of user 0 terminated abnormally with signal 6/ABRT, processing...
server # 2024-10-06T20:43:31.819972+00:00 server systemd-coredump[1503]: Process 1498 (indexer-worker) of user 0 terminated abnormally with signal 6/ABRT, processing...
server # [ 96.229291] systemd[1]: Started Process Core Dump (PID 1503/UID 0).
server # 2024-10-06T20:43:31.832374+00:00 server systemd[1]: Started Process Core Dump (PID 1503/UID 0).
server # [ 96.403356] rspamd[1448]: <cwfezm>; upstream; rspamd_upstream_fail: upstream 127.0.0.1 failed; reason: timeout waiting reply
server # 2024-10-06T20:43:32.006436+00:00 server rspamd[1448]: <cwfezm>; upstream; rspamd_upstream_fail: upstream 127.0.0.1 failed; reason: timeout waiting reply
server # [ 96.425373] systemd-coredump[1504]: Process 1498 (indexer-worker) of user 0 dumped core.
server #
server # Module libz.so.1 without build-id.
server # Module libicuio.so.74 without build-id.
server # Module libxapian.so.30 without build-id.
server # Module lib21_fts_xapian_plugin.so without build-id.
server # Module libgcc_s.so.1 without build-id.
server # Module libstdc++.so.6 without build-id.
server # Module libicudata.so.74 without build-id.
server # Module libicuuc.so.74 without build-id.
server # Module libicui18n.so.74 without build-id.
server # Module lib20_fts_plugin.so without build-id.
server # Module lib10_quota_plugin.so without build-id.
server # Module libdovecot.so.0 without build-id.
server # Module libdovecot-storage.so.0 without build-id.
server # Module indexer-worker without build-id.
server # Stack trace of thread 1502:
server # #0 0x00007f41b94477dc __pthread_kill_implementation (libc.so.6 + 0x927dc)
server # #1 0x00007f41b93f5516 raise (libc.so.6 + 0x40516)
server # #2 0x00007f41b93dd935 abort (libc.so.6 + 0x28935)
server # #3 0x00007f41b6df5c2b _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold (libstdc++.so.6 + 0xacc2b)
server # #4 0x00007f41b6e0520a _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6 + 0xbc20a)
server # #5 0x00007f41b6e05275 _ZSt9terminatev (libstdc++.so.6 + 0xbc275)
server # #6 0x00007f41b6e054c7 __cxa_throw (libstdc++.so.6 + 0xbc4c7)
server # #7 0x00007f41b6df8525 _ZSt24__throw_out_of_range_fmtPKcz.cold (libstdc++.so.6 + 0xaf525)
server # #8 0x00007f41b6c302b6 _ZN4XDoc14populate_stemsElPKc (lib21_fts_xapian_plugin.so + 0x332b6)
server # #9 0x00007f41b6c3095d _ZN11XDocsWriter6workerEv (lib21_fts_xapian_plugin.so + 0x3395d)
server # #10 0x00007f41b6e316d3 execute_native_thread_routine (libstdc++.so.6 + 0xe86d3)
server # #11 0x00007f41b9445a42 start_thread (libc.so.6 + 0x90a42)
server # #12 0x00007f41b94c505c __clone3 (libc.so.6 + 0x11005c)
server #
server # Stack trace of thread 1498:
server # #0 0x00007f41b948f535 clock_nanosleep@GLIBC_2.2.5 (libc.so.6 + 0xda535)
server # #1 0x00007f41b949a5c7 __nanosleep (libc.so.6 + 0xe55c7)
server # #2 0x00007f41b6c161e3 _ZL28fts_backend_xapian_unset_boxP18xapian_fts_backend.isra.0 (lib21_fts_xapian_plugin.so + 0x191e3)
server # #3 0x00007f41b6c16a9f _ZL26fts_backend_xapian_set_boxP18xapian_fts_backendP7mailbox (lib21_fts_xapian_plugin.so + 0x19a9f)
server # #4 0x00007f41b92dc1e6 fts_backend_update_deinit (lib20_fts_plugin.so + 0x4f1e6)
server # #5 0x00007f41b92e4c04 fts_transaction_end (lib20_fts_plugin.so + 0x57c04)
server # #6 0x00007f41b92e4dcb fts_transaction_commit (lib20_fts_plugin.so + 0x57dcb)
server # #7 0x00007f41b93a77bb quota_mailbox_transaction_commit (lib10_quota_plugin.so + 0x107bb)
server # #8 0x00007f41b97f773b mailbox_transaction_commit_get_changes (libdovecot-storage.so.0 + 0x6973b)
server # #9 0x00007f41b97f7841 mailbox_transaction_commit (libdovecot-storage.so.0 + 0x69841)
server # #10 0x000055d993df0f9b master_connection_input_args (indexer-worker + 0x3f9b)
server # #11 0x00007f41b96b0a4e connection_input_default (libdovecot.so.0 + 0x104a4e)
server # #12 0x00007f41b96d02e9 io_loop_call_io (libdovecot.so.0 + 0x1242e9)
server # #13 0x00007f41b96d1aba io_loop_handler_run_internal (libdovecot.so.0 + 0x125aba)
server # #14 0x00007f41b96d03a0 io_loop_handler_run (libdovecot.so.0 + 0x1243a0)
server # #15 0x00007f41b96d0570 io_loop_run (libdovecot.so.0 + 0x124570)
server # #16 0x00007f41b963ea03 master_service_run (libdovecot.so.0 + 0x92a03)
server # #17 0x000055d993df05eb main (indexer-worker + 0x35eb)
server # #18 0x00007f41b93df14e __libc_start_call_main (libc.so.6 + 0x2a14e)
server # #19 0x00007f41b93df209 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a209)
server # #20 0x000055d993df06a5 _start (indexer-worker + 0x36a5)
server # ELF object binary architecture: AMD x86-64
server #
server # 2024-10-06T20:43:32.028452+00:00 server systemd-coredump[1504]: Process 1498 (indexer-worker) of user 0 dumped core.#012#012Module libz.so.1 without build-id.#012Module libicuio.so.74 without build-id.#012Module libxapian.so.30 without build-id.#012Module lib21_fts_xapian_plugin.so without build-id.#012Module libgcc_s.so.1 without build-id.#012Module libstdc++.so.6 without build-id.#012Module libicudata.so.74 without build-id.#012Module libicuuc.so.74 without build-id.#012Module libicui18n.so.74 without build-id.#012Module lib20_fts_plugin.so without build-id.#012Module lib10_quota_plugin.so without build-id.#012Module libdovecot.so.0 without build-id.#012Module libdovecot-storage.so.0 without build-id.#012Module indexer-worker without build-id.#012Stack trace of thread 1502:#012#0 0x00007f41b94477dc __pthread_kill_implementation (libc.so.6 + 0x927dc)#012#1 0x00007f41b93f5516 raise (libc.so.6 + 0x40516)#012#2 0x00007f41b93dd935 abort (libc.so.6 + 0x28935)#012#3 0x00007f41b6df5c2b _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold (libstdc++.so.6 + 0xacc2b)#012#4 0x00007f41b6e0520a _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6 + 0xbc20a)#012#5 0x00007f41b6e05275 _ZSt9terminatev (libstdc++.so.6 + 0xbc275)#012#6 0x00007f41b6e054c7 __cxa_throw (libstdc++.so.6 + 0xbc4c7)#012#7 0x00007f41b6df8525 _ZSt24__throw_out_of_range_fmtPKcz.cold (libstdc++.so.6 + 0xaf525)#012#8 0x00007f41b6c302b6 _ZN4XDoc14populate_stemsElPKc (lib21_fts_xapian_plugin.so + 0x332b6)#012#9 0x00007f41b6c3095d _ZN11XDocsWriter6workerEv (lib21_fts_xapian_plugin.so + 0x3395d)#012#10 0x00007f41b6e316d3 execute_native_thread_routine (libstdc++.so.6 + 0xe86d3)#012#11 0x00007f41b9445a42 start_thread (libc.so.6 + 0x90a42)#012#12 0x00007f41b94c505c __clone3 (libc.so.6 + 0x11005c)#012#012Stack trace of thread 1498:#012#0 0x00007f41b948f535 clock_nanosleep@GLIBC_2.2.5 (libc.so.6 + 0xda535)#012#1 0x00007f41b949a5c7 __nanosleep (libc.so.6 + 0xe55c7)#012#2 0x00007f41b6c161e3 _ZL28fts_backend_xapian_unset_boxP18xapian_fts_backend.isra.0 (lib21_fts_xapian_plugin.so + 0x191e3)#012#3 0x00007f41b6c16a9f _ZL26fts_backend_xapian_set_boxP18xapian_fts_backendP7mailbox (lib21_fts_xapian_plugin.so + 0x19a9f)#012#4 0x00007f41b92dc1e6 fts_backend_update_deinit (lib20_fts_plugin.so + 0x4f1e6)#012#5 0x00007f41b92e4c04 fts_transaction_end (lib20_fts_plugin.so + 0x57c04)#012#6 0x00007f41b92e4dcb fts_transaction_commit (lib20_fts_plugin.so + 0x57dcb)#012#7 0x00007f41b93a77bb quota_mailbox_transaction_commit (lib10_quota_plugin.so + 0x107bb)#012#8 0x00007f41b97f773b mailbox_transaction_commit_get_changes (libdovecot-storage.so.0 + 0x6973b)#012#9 0x00007f41b97f7841 mailbox_transaction_commit (libdovecot-storage.so.0 + 0x69841)#012#10 0x000055d993df0f9b master_connection_input_args (indexer-worker + 0x3f9b)#012#11 0x00007f41b96b0a4e connection_input_default (libdovecot.so.0 + 0x104a4e)#012#12 0x00007f41b96d02e9 io_loop_call_io (libdovecot.so.0 + 0x1242e9)#012#13 0x00007f41b96d1aba io_loop_handler_run_internal (libdovecot.so.0 + 0x125aba)#012#14 0x00007f41b96d03a0 io_loop_handler_run (libdovecot.so.0 + 0x1243a0)#012#15 0x00007f41b96d0570 io_loop_run (libdovecot.so.0 + 0x124570)#012#16 0x00007f41b963ea03 master_service_run (libdovecot.so.0 + 0x92a03)#012#17 0x000055d993df05eb main (indexer-worker + 0x35eb)#012#18 0x00007f41b93df14e __libc_start_call_main (libc.so.6 + 0x2a14e)#012#19 0x00007f41b93df209 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a209)#012#20 0x000055d993df06a5 _start (indexer-worker + 0x36a5)#012ELF object binary architecture: AMD x86-64
weirdly rebuilding on 1.7.14 makes the issue disappear for me.
(nixos unstable on commit 915af63a65ff26a44d8897aa90bd9778e56ceb3a, happens with the automated tests of nixos-mailserver commit af7d3bf5daeba3fc28089b015c0dd43f06b176f2 )
grosjo
commented
1 month ago yes, this is a bug in the log debug. I fixed it. Can you try again ?
symphorien
commented
1 month ago 1.7.16 is fine, thanks.
grosjo
commented
1 month ago @Philippe23 please test with latest git to finalise the fix
Philippe23
commented
1 month ago With 9dff7eb1739ae98c8934923a85a2b78faaaad120
Program terminated with signal SIGSEGV, Segmentation fault.(gdb) bt
#0 0x00007f9faa725d28 in do_bswap (value=<optimized out>) at ./common/wordaccess.h:47
#1 do_unaligned_read<unsigned short> (ptr=0x9 <error: Cannot access memory at address 0x9>)
at ./common/wordaccess.h:122
#2 unaligned_read2 (ptr=0x9 <error: Cannot access memory at address 0x9>) at ./common/wordaccess.h:165
#3 Glass::DIR_END (b=0x0) at backends/glass/glass_table.h:114
#4 GlassTable::find_in_leaf (p=0x0, item=..., c=11, exact=exact@entry=@0x7f9fa9f7e9b7: false)
at backends/glass/glass_table.cc:428
#5 0x00007f9faa727d24 in GlassTable::find (this=this@entry=0x7f9f9c07e750, C_=C_@entry=0x7f9f9c07e818)
at backends/glass/glass_table.h:175
#6 0x00007f9faa72b3f1 in GlassTable::get_exact_entry (this=this@entry=0x7f9f9c07e750, key="Q189166", tag="")
at backends/glass/glass_table.cc:1489
#7 0x00007f9faa71d7cf in GlassPostListTable::merge_changes (this=0x7f9f9c07e750, term=..., changes=...)
at backends/glass/glass_postlist.cc:1213
#8 0x00007f9faa714353 in Inverter::flush_post_list (this=this@entry=0x7f9f9c07f350, table=..., term="Q189166")
at backends/glass/glass_inverter.cc:170
#9 0x00007f9faa7037f9 in GlassWritableDatabase::open_post_list (this=0x7f9f9c07e380, tname="Q189166")
at backends/glass/glass_database.cc:1502
#10 0x00007f9faa6987ad in Xapian::Database::Internal::replace_document (this=0x7f9f9c07e380, unique_term=...,
document=...) at backends/database.cc:197
#11 0x00007f9faa891451 in XDocsWriter::worker (this=0x55e434b65b10)
at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:980
#12 0x00007f9fac7ec6a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#13 0x00007f9facfea652 in ?? () from /lib64/libc.so.6
#14 0x00007f9fad0668ac in ?? () from /lib64/libc.so.6Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - adding 50001 terms to doc (Doc 189163 #lines=23 #stems=50001 status=2)
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=49851 status=1 : Populate 7 / 17 Header=XCT TextLength=26
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=49881 status=1 : Populate 6 / 17 Header=XCT TextLength=47
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=50001 status=1 : Populate 5 / 17 Header=XMID TextLength=61
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=50001 status=1 : Populate 4 / 17 Header=XLIST TextLength=20
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=50001 status=1 : Populate 3 / 17 Header=S TextLength=34
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=50001 status=1 : Populate 2 / 17 Header=XTO TextLength=35
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=50001 status=1 : Populate 1 / 17 Header=A TextLength=40
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189166 #lines=17 #stems=50001 status=1 : Done populating in 1029 ms (48591 stems/sec)
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: STEMS populated : 50001 [ XBDY//c XBDY//cl XBDY//cli XBDY//clic XBDY//click XBDY//click_ XBDY//click_e XBDY//click_e_ XBDY//click_e_i XBDY//click_e_in XCTxt/plain XCTxt/plai XCTxt/pla XCTxt/pl XCTxt/p XCTxt/html XCTxt/htm XCTxt/ht XCTxt/h]
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Creating Xapian doc : Doc 189166 #lines=17 #stems=50001 status=2
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - adding 50001 terms to doc (Doc 189166 #lines=17 #stems=50001 status=2)
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - create_doc done (Doc 189163 #lines=23 #stems=50001 status=2)
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Pushing : Doc 189163 #lines=23 #stems=50001 status=3
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Replace doc : Doc 189163 #lines=23 #stems=50001 status=3 Free memory : 1024MB
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - create_doc done (Doc 189166 #lines=17 #stems=50001 status=2)
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Pushing : Doc 189166 #lines=17 #stems=50001 status=3
Oct 18 14:51:50 myhostname dovecot[26265]: indexer-worker(USER1)<27044><kXxMw8Qk8dAmTRZ9:MbMWMLutEmekaQAA0J78UA>: FTS Xapian: New doc incoming (#190258)
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc done
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Searching doc
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Populating stems : Doc 189167 #lines=20 #stems=0 status=1
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=0 status=1 : Populate 20 headers with strings
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=0 status=1 : Populate 20 / 20 Header=XBDY TextLength=236
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Replace doc : Doc 189166 #lines=17 #stems=50001 status=3 Free memory : 1024MB
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=891 status=1 : Populate 19 / 20 Header=XBDY TextLength=392
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1066 status=1 : Populate 18 / 20 Header=XBDY TextLength=172
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1071 status=1 : Populate 17 / 20 Header=XBDY TextLength=291
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1205 status=1 : Populate 16 / 20 Header=XBDY TextLength=226
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1251 status=1 : Populate 15 / 20 Header=XBDY TextLength=661
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1484 status=1 : Populate 14 / 20 Header=XBDY TextLength=0
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1484 status=1 : Populate 13 / 20 Header=XBDY TextLength=6
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1484 status=1 : Populate 12 / 20 Header=XCT TextLength=14
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=1511 status=1 : Populate 11 / 20 Header=XCT TextLength=489
Oct 18 14:51:50 myhostname xapian-docswriter[27044]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189167 #lines=20 #stems=9187 status=1 : Populate 10 / 20 Header=XBDY TextLength=3810
Oct 18 14:51:51 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Can't write doc2 : std::exception
Oct 18 14:51:51 myhostname dovecot[26265]: indexer-worker(USER1)<27044><kXxMw8Qk8dAmTRZ9:MbMWMLutEmekaQAA0J78UA>: FTS Xapian: Previous doc ready to index (#190257)
Oct 18 14:51:51 myhostname dovecot[26265]: indexer-worker(USER1)<27044><kXxMw8Qk8dAmTRZ9:MbMWMLutEmekaQAA0J78UA>: FTS Xapian: Start indexing #190258 (Trash) : Queue size = 1077
Oct 18 14:51:51 myhostname xapian-docswriter[27044]: FTS Xapian: Can not allocate ICU translator + FreeMem=1024MB
Oct 18 14:51:51 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Pushing : Doc 189166 #lines=17 #stems=50001 status=3
Oct 18 14:51:51 myhostname xapian-docswriter[27044]: FTS Xapian: Can not allocate ICU translator + FreeMem=1024MB
Oct 18 14:51:51 myhostname xapian-docswriter[27044]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Replace doc : Doc 189166 #lines=17 #stems=50001 status=3 Free memory : 1024MB
Oct 18 14:51:51 myhostname kernel: indexer-worker[27045]: segfault at 9 ip 00007f9faa725d28 sp 00007f9fa9f7e960 error 4 in libxapian.so.30.12.6[7f9faa616000+1ba000] likely on CPU 0 (core 0, socket 0)
Oct 18 14:51:51 myhostname kernel: Code: 5c e9 bc 12 ef ff 0f 1f 40 00 5b 5d 41 5c c3 90 66 90 f3 0f 1e fa 41 57 49 89 ff 41 56 41 55 41 54 49 89 f4 55 53 48 83 ec 18 <0f> b7 6f 09 48 89 4c 24 08 66 c1 c5 08 0f b7 ed 83 fa ff 0f 84 ff
Oct 18 14:52:16 myhostname dovecot[26265]: imap-login: Login: user=<USER2>, method=PLAIN, rip=107.222.116.230, lip=172.30.3.214, mpid=1616, TLS, session=<6W3SzMQkgedr3nTm>
Oct 18 14:52:16 myhostname dovecot[26265]: imap-login: Login: user=<USER2>, method=PLAIN, rip=107.222.116.230, lip=172.30.3.214, mpid=1617, TLS, session=<nnjSzMQkhOdr3nTm>
Oct 18 14:52:16 myhostname dovecot[26265]: imap-login: Login: user=<USER2>, method=PLAIN, rip=107.222.116.230, lip=172.30.3.214, mpid=1618, TLS, session=<Vm7SzMQkg+dr3nTm>
Oct 18 14:52:16 myhostname dovecot[26265]: imap-login: Login: user=<USER2>, method=PLAIN, rip=107.222.116.230, lip=172.30.3.214, mpid=1619, TLS, session=<6W3SzMQkgudr3nTm>
Oct 18 14:52:30 myhostname dovecot[26265]: imap(USER2)<1617><nnjSzMQkhOdr3nTm>: FTS Xapian: Starting with partial=3 full=20 verbose=1 max_threads=2 lowmemory=500 MB vs freemem=256 MB
Oct 18 14:52:30 myhostname dovecot[26265]: imap(USER2)<1619><6W3SzMQkgudr3nTm>: FTS Xapian: Starting with partial=3 full=20 verbose=1 max_threads=2 lowmemory=500 MB vs freemem=256 MB
Oct 18 14:52:30 myhostname dovecot[26265]: imap(USER2)<1618><Vm7SzMQkg+dr3nTm>: FTS Xapian: Starting with partial=3 full=20 verbose=1 max_threads=2 lowmemory=500 MB vs freemem=256 MB
Oct 18 14:52:30 myhostname dovecot[26265]: imap(USER2)<1616><6W3SzMQkgedr3nTm>: FTS Xapian: Starting with partial=3 full=20 verbose=1 max_threads=2 lowmemory=500 MB vs freemem=256 MB
Oct 18 14:52:31 myhostname postfix/anvil[17560]: statistics: max connection rate 1/60s for (smtp:194.39.206.151) at Oct 18 14:45:17
Oct 18 14:52:31 myhostname postfix/anvil[17560]: statistics: max connection count 1 for (smtp:194.39.206.151) at Oct 18 14:45:17
Oct 18 14:52:31 myhostname postfix/anvil[17560]: statistics: max cache size 4 at Oct 18 14:45:50
Oct 18 14:52:34 myhostname dovecot[26265]: indexer-worker(USER1)<27044><kXxMw8Qk8dAmTRZ9:MbMWMLutEmekaQAA0J78UA>: Fatal: master: service(indexer-worker): child 27044 killed with signal 11 (core dumped)FTS Xapian: Can not allocate ICU translator + FreeMem=1024MB
That seems fishy. 1G of FreeMem feels like it should be more than enough space.
(That said, I still say it feels like Xapian's GlassTable library shouldn't be crashing.)
grosjo
commented
1 month ago My measurement of memory seemed to be wrong. Please try again latest git
Philippe23
commented
1 month ago I'm also seeing crashes that are complaining about a "Leaked t_pop() call":
Oct 20 13:40:50 myhostname dovecot[21190]: indexer-worker(USER1)<29686><apGGnuskFvcmTRZ9:GDf3LIxAFWf2cwAA0J78UA>: Warning: FTS Xapian: Available memory 222 MB
Oct 20 13:40:50 myhostname xapian-docswriter[29686]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Replace doc : Doc 189133 #lines=6 #stems=1939 status=3 Free memory : 222MB
Oct 20 13:40:50 myhostname xapian-docswriter[29686]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc done
Oct 20 13:40:50 myhostname xapian-docswriter[29686]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Searching doc
Oct 20 13:40:50 myhostname xapian-docswriter[29686]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Populating stems : Doc 189135 #lines=6 #stems=0 status=1
Oct 20 13:40:50 myhostname xapian-docswriter[29686]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189135 #lines=6 #stems=0 status=1 : Populate 6 headers with strings
Oct 20 13:40:50 myhostname xapian-docswriter[29686]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189135 #lines=6 #stems=0 status=1 : Populate 6 / 6 Header=XBDY TextLength=1472
Oct 20 13:40:50 myhostname dovecot[21190]: indexer-worker(USER1)<29686><apGGnuskFvcmTRZ9:GDf3LIxAFWf2cwAA0J78UA>: Warning: FTS Xapian: Available memory 222 MB
Oct 20 13:40:50 myhostname dovecot[21190]: indexer-worker(USER1)<29686><apGGnuskFvcmTRZ9:GDf3LIxAFWf2cwAA0J78UA>: Warning: FTS Xapian: Available memory 222 MB
Oct 20 13:40:50 myhostname dovecot[21190]: indexer-worker(USER1)<29686><apGGnuskFvcmTRZ9:GDf3LIxAFWf2cwAA0J78UA>: Panic: Leaked t_pop() call
Oct 20 13:40:50 myhostname dovecot[21190]: indexer-worker(USER1)<29686><apGGnuskFvcmTRZ9:GDf3LIxAFWf2cwAA0J78UA>: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7fba981e9876] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7fba981e99c2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x105d77) [0x7fba981f6d77] -> /usr/lib64/dovecot/libdovecot.so.0(+0x105e17) [0x7fba981f6e17] -> /usr/lib64/dovecot/libdovecot.so.0(+0x572f7) [0x7fba981482f7] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xab3d) [0x7fba97eccb3d] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x13f1d) [0x7fba97ed5f1d] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x32) [0x7fba9831fff2] -> /usr/libexec/dovecot/indexer-worker(+0x3031) [0x55da0a94d031] -> /usr/lib64/dovecot/libdovecot.so.0(connection_input_default+0x1c6) [0x7fba981ee5d6] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x7fba9820d34d] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x13a) [0x7fba9820ea4a] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x7fba9820d3f4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7fba9820d5b0] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x17) [0x7fba9817e7d7] -> /usr/libexec/dovecot/indexer-worker(main+0xeb) [0x55da0a94ca5b] -> /lib64/libc.so.6(+0x260d0) [0x7fba97f2f0d0] -> /lib64/libc.so.6(__libc_start_main+0x89) [0x7fba97f2f189] -> /usr/libexec/dovecot/indexer-worker(_start+0x25) [0x55da0a94cb15]
Oct 20 13:40:51 myhostname dovecot[21190]: indexer-worker(USER1)<29686><apGGnuskFvcmTRZ9:GDf3LIxAFWf2cwAA0J78UA>: Fatal: master: service(indexer-worker): child 29686 killed with signal 6 (core dumped)(3rd to last line -- was trying to give some context in case it helps.)
This is on e205535fdd49b20782e54a540643bdf7eafe94cb
Philippe23
commented
1 month ago And crashes when you're logging the free memory ... ?
(gdb) bt
#0 0x00007f641f42935c in ?? () from /lib64/libc.so.6
#1 0x00007f641f3d91f6 in raise () from /lib64/libc.so.6
#2 0x00007f641f3c18f7 in abort () from /lib64/libc.so.6
#3 0x00007f641f5dc64d in ?? () from /usr/lib64/dovecot/libdovecot.so.0
#4 0x00007f641f68ae17 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
#5 0x00007f641f5dc2f7 in i_panic () from /usr/lib64/dovecot/libdovecot.so.0
#6 0x00007f641f5dc32d in ?? () from /usr/lib64/dovecot/libdovecot.so.0
#7 0x00007f641f68ab2e in ?? () from /usr/lib64/dovecot/libdovecot.so.0
#8 0x00007f641f68aeda in i_warning () from /usr/lib64/dovecot/libdovecot.so.0
#9 0x00007f641ccaf211 in fts_backend_xapian_get_free_memory () at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:51
#10 0x00007f641ccce9ea in XDocsWriter::worker (this=0x55a1232ad740) at /usr/src/debug/net-mail/dovecot-fts-xapian-9999/dovecot-fts-xapian-9999/src/fts-backend-xapian-functions.cpp:968
#11 0x00007f641ec296a3 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6
#12 0x00007f641f427652 in ?? () from /lib64/libc.so.6
#13 0x00007f641f4a38ac in ?? () from /lib64/libc.so.6if(fts_xapian_settings.verbose>0) i_warning("FTS Xapian: Available memory %ld MB",long(m/1024.0));Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: FTS Xapian: Start indexing #189010 (Trash) : Queue size = 2
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=4427 status=1 : Populate 7 / 18 Header=XCT TextLength=42
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=4478 status=1 : Populate 6 / 18 Header=XCT TextLength=63
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=5009 status=1 : Populate 5 / 18 Header=XLIST TextLength=54
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=5489 status=1 : Populate 4 / 18 Header=XMID TextLength=83
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=5491 status=1 : Populate 3 / 18 Header=XTO TextLength=33
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #1 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189008 #lines=10 #stems=3871 status=1 : Populate 7 / 10 Header=XBDY TextLength=743
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=5898 status=1 : Populate 2 / 18 Header=A TextLength=28
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: FTS Xapian: New doc incoming (#189011)
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: FTS Xapian: Previous doc ready to index (#189010)
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: FTS Xapian: Start indexing #189011 (Trash) : Queue size = 3
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=6051 status=1 : Populate 1 / 18 Header=S TextLength=26
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Doc 189007 #lines=18 #stems=6064 status=1 : Done populating in 103 ms (58873 stems/sec)
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: STEMS populated : 6064 [ A_co A_com A_com_ A_in A_inf A_info A_info_ A_info_m A_info_mi A_info_mir XTOxshift_com_ XTOxshift_com XTOxshift_co XTOxshift_c XTOxshift_ XTOxshift XTOxshif XTOxshi XTOxsh]
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Creating Xapian doc : Doc 189007 #lines=18 #stems=6064 status=2
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - adding 6064 terms to doc (Doc 189007 #lines=18 #stems=6064 status=2)
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - create_doc done (Doc 189007 #lines=18 #stems=6064 status=2)
Oct 20 13:38:00 myhostname xapian-docswriter[29280]: DW #2 (Trash,/home/USER1/.maildir/xapian-indexes/db_e0e0872a837b795b6d200000d09efc50) - Pushing : Doc 189007 #lines=18 #stems=6064 status=3
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: Warning: FTS Xapian: Available memory 253 MB
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: Panic: Leaked t_pop() call
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: Panic: Leaked t_pop() call
Oct 20 13:38:00 myhostname dovecot[21190]: indexer-worker(USER1)<29280><qRntm+skYPYmTRZ9:9rb7EfY/FWdgcgAA0J78UA>: Fatal: master: service(indexer-worker): child 29280 killed with signal 6 (core dumped)@Philippe23 it does not seem you use the latest git
Philippe23
commented
1 month ago It was latest 2 days ago when you asked. 😆
But I updated again earlier today and told it to re-index. I'll report back.
kshigapov
commented
4 weeks ago Hello!
First of all, thx for your work!
I'm trying to use your plugin on Rocky Linux 9.4 with no luck on some mail boxes which size is > 3.5 GB
After entering command dovecot index -u $some_mail_box '*' after some time it fails with:
Segmentation fault (core dumped)
This mail box size is 11.03 GB
coredumpctl list :
Sun 2024-10-27 17:52:51 UTC 1498943 2000 2000 SIGSEGV present /usr/bin/doveadm 162.7M
coredumpctl gdb 1498943 :
Stack trace of thread 1498950:
#0 0x00007f612c27dbe4 _IO_feof (libc.so.6 + 0x7dbe4)
#1 0x00007f612bf7b6b4 fts_backend_xapian_get_free_memory (lib21_fts_xapian_plugin.so + 0x136b4)
#2 0x00007f612bf9835c _ZN11XDocsWriter6workerEv (lib21_fts_xapian_plugin.so + 0x3035c)
#3 0x00007f6129adbad4 execute_native_thread_routine (libstdc++.so.6 + 0xdbad4)
#4 0x00007f612c289c02 start_thread (libc.so.6 + 0x89c02)
#5 0x00007f612c30ec40 __clone3 (libc.so.6 + 0x10ec40)
...
Core was generated by 'dovecot index -u $some_mail_box *'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 _IO_feof (fp=0x0) at feof.c:35
35 if (!_IO_need_lock (fp))dovecot.log stops on :
Mailbox opened because: SELECT
Waiting for dict to finish pending operationsin dovecot.conf :
plugin {
...
fts = xapian
fts_xapian = partial=3 full=20
fts_autoindex = yes
fts_enforced = yes
fts_autoindex_exclude = \Trash
fts_autoindex_exclude2 = \Junk
}
service indexer-worker {
vsz_limit = 2G
process_limit = 0
}I see *.db files in xapian-indexes folder in $some_mail_box But search in roundcube fails with timeout
Server info: 2 vcpu, 8 gb ram
What more useful info I can give you?
grosjo
commented
4 weeks ago maybe un " bt full" on the coredump ?
grosjo
commented
4 weeks ago Please use latest git
Philippe23
commented
4 weeks ago Figured I'd report back in, as it's been about a week. I'm still running 17da18efc8c2c1f444cfa93825afed00156f5ef6 ("latest" on Oct. 20th), but have not yet had a crash.
kshigapov
commented
4 weeks ago Checked latest git. No more crashes.
There was error failed: Too many open files after doveadm index command.
Setting ulimit -n 166384 fixed it.
Now search works as expected.
Thank you!
grosjo
commented
2 weeks ago I close this one. Do not hesitate to re-open if ever
Philippe23
commented
2 weeks ago I still haven't had another crash. I was just thinking about suggesting closing this yesterday. 👍
Thanks for all the work on this. I know these are hard to track down.
I'm seeing some core dumps from 1.7.14.
Logs that reference the PID from the core:
A different trace:
Logs:
Another one:
Logs:
I also saw this once in my logs, but didn't find a core for the specific process: