search

grosjo / fts-xapian

Dovecot FTS plugin based on Xapian
GNU Lesser General Public License v2.1
91 stars 19 forks source link

Crash while indexing with "std::exception" on aarch64 #158

Closed Grokenberger closed 3 months ago

Grokenberger commented 3 months ago

I'm aiming to migrate from Arch Linux x86_64 to ArchLinuxArm aarch64 with pretty much identical software versions:

Name            : dovecot
Version         : 2.3.20-4
Architecture    : aarch64
Packager        : Arch Linux ARM Build System <builder+n1@archlinuxarm.org>
--
Name            : xapian-core
Version         : 1:1.4.25-1
Architecture    : aarch64
Packager        : Arch Linux ARM Build System <builder+n1@archlinuxarm.org>
--
Name            : dovecot-fts-xapian
Version         : 1.7.10-1
Architecture    : aarch64
Packager        : Arch Linux ARM Build System <builder+seattle@archlinuxarm.org>

Related settings for dovecot:

plugin {
  fts = xapian
  fts_xapian = partial=4 full=20 verbose=0
  fts_autoindex = yes
  fts_enforced = yes
  fts_autoindex_exclude = \Trash
  fts_languages = en de
  fts_language_config = /usr/share/libexttextcat/fpdb.conf
  fts_header_includes = X-Spam-Status
  fts_header_excludes = Received DKIM-* X-*
  fts_tokenizers = generic email-address # default
  fts_tokenizer_generic = algorithm=simple maxlen=30
}

Replication went fine, but with each indexing attempt on aarch64 I got

May 20 19:12:50 larry dovecot[1434]: indexer-worker(hugo@example.net)<10227><SL4hH36ES2btJwAA0J78UA:CfWdJX6ES2bzJwAA0J78UA>: FTS Xapian: Waiting for all pending documents to be processed (Sleep5)
May 20 19:46:08 larry xapian-docswriter[10227]: DW #46 (INBOX,/home/vmail/example.net/hugo/Maildir/xapian-indexes/db_3d6c3705f9dae358156600001a6400fe) - Can't add document2 : std::exception
May 20 19:46:08 larry xapian-docswriter[10227]: DW #46 (INBOX,/home/vmail/example.net/hugo/Maildir/xapian-indexes/db_3d6c3705f9dae358156600001a6400fe) -  Retrying (/home/vmail/example.net/hugo/Maildir/xapian-indexes/db_3d6c3705f9dae3581>
May 20 19:46:31 larry dovecot[1434]: indexer-worker(hugo@example.net)<10227><SL4hH36ES2btJwAA0J78UA:CfWdJX6ES2bzJwAA0J78UA>: Fatal: master: service(indexer-worker): child 10227 killed with signal 11 (core dumped)

The core dump looks like this:

           PID: 10227 (indexer-worker)
           UID: 0 (root)
           GID: 5000 (vmail)
        Signal: 11 (SEGV)
     Timestamp: Mon 2024-05-20 19:46:08 CEST (16h ago)
  Command Line: dovecot/indexer-worker
    Executable: /usr/lib/dovecot/indexer-worker
 Control Group: /system.slice/dovecot.service
          Unit: dovecot.service
         Slice: system.slice
       Boot ID: 980273f071e0440cbb8fe187d7c5d471
    Machine ID: 793ff9786b1b40e3b11d06dc121121c0
      Hostname: larry
       Storage: /var/lib/systemd/coredump/core.indexer-worker.0.980273f071e0440cbb8fe187d7c5d471.10227.1716227168000000.zst (present)
  Size on Disk: 243.1M
       Message: Process 10227 (indexer-worker) of user 0 dumped core.

                Stack trace of thread 10274:
                #0  0x0000ffff82f808bc n/a (libxapian.so.30 + 0x1208bc)
                #1  0x0000ffff82f7fbec n/a (libxapian.so.30 + 0x11fbec)
                #2  0x0000ffff82f71f34 n/a (libxapian.so.30 + 0x111f34)
                #3  0x0000ffff82f697f8 n/a (libxapian.so.30 + 0x1097f8)
                #4  0x0000ffff82f59124 n/a (libxapian.so.30 + 0xf9124)
                #5  0x0000ffff82f5cdf0 n/a (libxapian.so.30 + 0xfcdf0)
                #6  0x0000ffff82eb23f8 _ZN6Xapian16WritableDatabase6commitEv (libxapian.so.30 + 0x523f8)
                #7  0x0000ffff830f41dc _ZN11XDocsWriter6workerEv (lib21_fts_xapian_plugin.so + 0x341dc)
                #8  0x0000ffff8504479c execute_native_thread_routine (libstdc++.so.6 + 0xe479c)
                #9  0x0000ffff8596a624 n/a (libc.so.6 + 0x8a624)
                #10 0x0000ffff859d064c n/a (libc.so.6 + 0xf064c)

                Stack trace of thread 10271:
                #0  0x0000ffff8597ff3c memmove (libc.so.6 + 0x9ff3c)
                #1  0x0000ffff830f26a8 _ZN6XNGram8add_stemEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x326a8)
                #2  0x0000ffff830f2d4c _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32d4c)
                #3  0x0000ffff830f2e84 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e84)
                #4  0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #5  0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #6  0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #7  0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #8  0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #9  0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #10 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #11 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #12 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #13 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #14 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #15 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #16 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #17 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #18 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #19 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #20 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #21 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #22 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #23 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #24 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #25 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #26 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #27 0x0000ffff830f2e78 _ZN6XNGram3addEPN6icu_7413UnicodeStringE (lib21_fts_xapian_plugin.so + 0x32e78)
                #28 0x0000ffff830f3850 _ZN4XDoc14populate_stemsElPKc (lib21_fts_xapian_plugin.so + 0x33850)
                #29 0x0000ffff830f3dc0 _ZN11XDocsWriter6workerEv (lib21_fts_xapian_plugin.so + 0x33dc0)
                #30 0x0000ffff8504479c execute_native_thread_routine (libstdc++.so.6 + 0xe479c)
                #31 0x0000ffff8596a624 n/a (libc.so.6 + 0x8a624)
                #32 0x0000ffff859d064c n/a (libc.so.6 + 0xf064c)

                Stack trace of thread 10227:
                #0  0x0000ffff859a0680 clock_nanosleep (libc.so.6 + 0xc0680)
                #1  0x0000ffff859aae9c __nanosleep (libc.so.6 + 0xcae9c)
                #2  0x0000ffff830d458c n/a (lib21_fts_xapian_plugin.so + 0x1458c)
                #3  0x0000ffff830d4d9c n/a (lib21_fts_xapian_plugin.so + 0x14d9c)
                #4  0x0000ffff857f88b0 fts_backend_update_deinit (lib20_fts_plugin.so + 0x88b0)
                #5  0x0000ffff858030b8 n/a (lib20_fts_plugin.so + 0x130b8)
                #6  0x0000ffff85803330 n/a (lib20_fts_plugin.so + 0x13330)
                #7  0x0000ffff85852308 n/a (lib15_notify_plugin.so + 0x2308)
                #8  0x0000ffff8587fa68 n/a (lib10_quota_plugin.so + 0xfa68)
                #9  0x0000ffff85cf05ac mailbox_transaction_commit_get_changes (libdovecot-storage.so.0 + 0x505ac)
                #10 0x0000ffff85cf0728 mailbox_transaction_commit (libdovecot-storage.so.0 + 0x50728)
                #11 0x0000aaaadbb229f4 n/a (indexer-worker + 0x29f4)
                #12 0x0000ffff85ba6030 connection_input_default (libdovecot.so.0 + 0x106030)
                #13 0x0000ffff85bcccf4 io_loop_call_io (libdovecot.so.0 + 0x12ccf4)
                #14 0x0000ffff85bced68 io_loop_handler_run_internal (libdovecot.so.0 + 0x12ed68)
                #15 0x0000ffff85bcce48 io_loop_handler_run (libdovecot.so.0 + 0x12ce48)
                #16 0x0000ffff85bcd160 io_loop_run (libdovecot.so.0 + 0x12d160)
                #17 0x0000ffff85b1bb94 master_service_run (libdovecot.so.0 + 0x7bb94)
                #18 0x0000aaaadbb220a8 main (indexer-worker + 0x20a8)
                #19 0x0000ffff85908294 n/a (libc.so.6 + 0x28294)
                #20 0x0000ffff85908378 __libc_start_main (libc.so.6 + 0x28378)
                #21 0x0000aaaadbb221b0 _start (indexer-worker + 0x21b0)
                ELF object binary architecture: AARCH64

GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/dovecot/indexer-worker...
(No debugging symbols found in /usr/lib/dovecot/indexer-worker)
[New LWP 10274]
[New LWP 10271]
[New LWP 10227]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `dovecot/indexer-worker'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000ffff82f808bc in ?? () from /usr/lib/libxapian.so.30
[Current thread is 1 (Thread 0xffff81e821a0 (LWP 10274))]

and the "bt full" command like this:

#0  0x0000ffff82f808bc in ?? () from /usr/lib/libxapian.so.30
No symbol table info available.
#1  0x0000ffff82f7fbec in ?? () from /usr/lib/libxapian.so.30
No symbol table info available.
#2  0x0000ffff82f71f34 in ?? () from /usr/lib/libxapian.so.30
No symbol table info available.
#3  0x0000ffff82f697f8 in ?? () from /usr/lib/libxapian.so.30
No symbol table info available.
#4  0x0000ffff82f59124 in ?? () from /usr/lib/libxapian.so.30
No symbol table info available.
#5  0x0000ffff82f5cdf0 in ?? () from /usr/lib/libxapian.so.30
No symbol table info available.
#6  0x0000ffff82eb23f8 in Xapian::WritableDatabase::commit() () from /usr/lib/libxapian.so.30
No symbol table info available.
#7  0x0000ffff830f41dc in XDocsWriter::worker() () from /usr/lib/dovecot/modules/lib21_fts_xapian_plugin.so
No symbol table info available.
#8  0x0000ffff8504479c in std::execute_native_thread_routine (__p=0xaaaaecf36510) at /build/gcc/src/gcc/libstdc++-v3/src/c++11/thread.cc:104
        __t = <optimized out>
#9  0x0000ffff8596a624 in ?? () from /usr/lib/libc.so.6
No symbol table info available.
#10 0x0000ffff859d064c in ?? () from /usr/lib/libc.so.6
No symbol table info available.

Hope, I gathered all information you need. Indexing on the old x64_64 machine works without any issue.

grosjo commented 3 months ago

Hi how much memory is on the machine and what is your VZ value in dovecot.conf ?

Grokenberger commented 3 months ago

Hi how much memory is on the machine and what is your VZ value in dovecot.conf ?

Machine has 8G RAM, both indexer-worker and default_vsz_limit is set to 4G.

grosjo commented 3 months ago

Ok, that shall be fine. Have you tried the same with latest git of the plugin ?

Grokenberger commented 3 months ago

No further crash. But some weird log lines. Need some time to investigate. Thanks deeply so far!