Panic with flatcure, virtual mailboxes: file index-search-result.c: line 132

[doits]

I have to following panics with flatcurve enabled:

Panic line 132

``` Panic: file index-search-result.c: line 132 (index_search_result_update_flags): assertion failed: (result->search_args->args == &search_arg) Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x74e9890a2846] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x74e9890a2962] -> /usr/lib/dovecot/libdovecot.so.0(+0x10d81b) [0x74e9890af81b] -> /usr/lib/dovecot/libdovecot.so.0(+0x10d8b7) [0x74e9890af8b7] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e2e2) [0x74e9890002e2] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x518a4) [0x74e9891d38a4] -> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x274e) [0x74e988bf8a9e] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x5c) [0x74e9891eff9c] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x39) [0x74e9891f0039] -> dovecot/imap(cmd_select_full+0x1df) [0x62354bd8326f] -> dovecot/imap(command_exec+0xa4) [0x62354bd8b444] -> dovecot/imap(+0x22392) [0x62354bd89392] -> dovecot/imap(+0x22444) [0x62354bd89444] -> dovecot/imap(client_handle_input+0x1bd) [0x62354bd8989d] -> dovecot/imap(client_input+0x74) [0x62354bd89e64] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x74e9890c603d] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x13a) [0x74e9890c77aa] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x74e9890c60e4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x74e9890c62a0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x17) [0x74e989036ad7] -> dovecot/imap(main+0x570) [0x62354bd7ad20] -> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x74e988c29d90] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x74e988c29e40] -> dovecot/imap(_start+0x25) [0x62354bd7ade5] ```

It happens when accessing some virtual mailboxes.

Example of dovecot-virtual virtual mailboxes that produce this error:

something-*/excep-dev*
  header x-abc-exception "1"

something/mailbox
  or or
    subject "Subject 1"
    subject "Another one"
    subject "Third"

Example where it works:

Inbox
some/mailbox
another/one
abc/def

some-prefix-*/folder

When I comment out fts = flatcurve in dovecot configuration, the error is gone.

Configuration of flatcurve:

plugin {
  fts = flatcurve

  fts_autoindex = yes

  fts_tokenizer_generic = algorithm=simple
  fts_tokenizers = generic email-address

  fts_languages = de en

  fts_filters = normalizer-icu snowball stopwords
  fts_filters_en = lowercase snowball stopwords
}

Configuration of virtual folders:

namespace {
  prefix = _/
  separator = /
  location = virtual:~/virtual:LAYOUT=maildir++
}

Error feels similar to #61.

Dovecot 2.3.21 (47349e2482) and flatcure compiled from 0748ec9c4949275ad2dc6a6979ff2cc404fc5187.

Can you maybe reproduce the problem with this information? How can we debug this further?

[doits]

One thing I noticed which maybe helps: When I delete the dovecot.index{,.log} files and access the virtual folder, the first access is successful and it displays the messages according to the imap filter. The dovecot.index is created on access. But accessing the folder a second time leads to the panic mentioned above.

[doits]

Sorry, one more information, actually it is like this:

• Removing dovecot.index{,.log} files allows to access the virtual folder
• dovecot.index gets recreated on access and the folder can be accessed multiple times
• Marking an e-mail read triggers the error, though! Now each next access raises the error
• Removing dovecot.index{,.log} restores access again and the e-mails are still marked as read (so this information is not lost), until the next e-mail is marked as read: then the folder cannot be accessed again

Disabling indexes with location = virtual:~/virtual:LAYOUT=maildir++:INDEX=MEMORY is a work around and fixes the error, though I get a lot of different imap: Error: terminate called after throwing an instance of 'std::bad_alloc' when accessing the folders (I guess out of memory or something), and one of my email clients (thunderbird) redownloads the whole folder on every access – so no usable workaround

And I got another panic on file index-search-result.c: line 174 (index_search_result_update_appends): assertion failed: (result->search_args->args == &search_arg), which seems to be the same assertion.

Backtrace

``` Panic: file index-search-result.c: line 174 (index_search_result_update_appends): assertion failed: (result->search_args->args == &search_arg) Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x791235dfa846] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x791235dfa962] -> /usr/lib/dovecot/libdovecot.so.0(+0x10d81b) [0x791235e0781b] -> /usr/lib/dovecot/libdovecot.so.0(+0x10d8b7) [0x791235e078b7] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e2e2) [0x791235d582e2] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x518f0) [0x791235f2b8f0] -> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x276a) [0x791235c58aba] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x5c) [0x791235f47f9c] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x39) [0x791235f48039] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x37) [0x791235fc5467] -> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0xc6c5) [0x791235c536c5] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x145e2) [0x791235c9a5e2] -> /usr/lib/dovecot/modules/lib01_acl_plugin.so(+0x10cb9) [0x791235ce5cb9] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x64) [0x791235f485a4] -> dovecot/imap(imap_status_get+0x98) [0x623756c8c748] -> dovecot/imap(cmd_status+0x19e) [0x623756c7ca7e] -> dovecot/imap(command_exec+0xa4) [0x623756c83444] -> dovecot/imap(+0x22392) [0x623756c81392] -> dovecot/imap(+0x22444) [0x623756c81444] -> dovecot/imap(client_handle_input+0x1bd) [0x623756c8189d] -> dovecot/imap(client_input+0x74) [0x623756c81e64] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x791235e1e03d] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x13a) [0x791235e1f7aa] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x791235e1e0e4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x791235e1e2a0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x17) [0x791235d8ead7] -> dovecot/imap(main+0x570) [0x623756c72d20] -> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x791235a29d90] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x791235a29e40] -> dovecot/imap(_start+0x25) [0x623756c72de5] ```

[doits]

Is there anything I can do to help you debug it? Can you reproduce it? AFAIK to reproduce it, it should be enough to create some virtual folders with the above imap filters.

[slusarz]

I haven't been able to look at this yet. Maybe soon.

[slusarz]

This is not a flatcurve bug - it is a bug within Dovecot's indexing/fts handling.

(Detailed debug below, so that core team can understand the problem.)

As discussed above, this bug is triggered with virtual mailboxes on subsequent accesses. The initial access will work fine; the assert crash only happens when the Dovecot lib-storage index code reads the indexes created for the virtual mailboxes.

index_search_result_update_appends() in lib-storage/index/index-search-result.c

• Inside this function, a "temporary search parameter to limit the search only to the new messages" is added to the mail search args. Later, it is checked that the mail_search args points to this same search parameter after the search is executed. It is this assert check that is failing, because the mail args is changing during the course of the search.
• i_assert(result->search_args->args == &search_arg);

The search args are being altered in plugins/fts/fts-search.args.c, fts_search_args_expand(). Specifically, in that function, either fts_search_args_expand_language_top_level() or fts_search_args_expand_tree() is being called, which can replace the current args parameter with a new pointer.

fts_search_args_expand() is being called in plugins//fts/fts-search.c by fts_search_try_lookup() when the FTS backend has the FTS_BACKEND_FLAG_TOKENIZED_INPUT flag set. Flatcurve has this flag set.

This flag is defined in plugins/fts/fts-api-private.h. Comments say:

        /* Tokenize all the input. update_build_more() will be called a single
           directly indexable token at a time. Searching will modify the search
           args so that lookup() sees only tokens that can be directly
           searched. */

Thus, the comments explicitly say the search args will be modified if this flag is set. This modification is incompatible with the assert check in index_search_result_update_appends(), which requires that the search args not be modified.

The core team will have to determine whether this behavior is correct, or if the assert check is invalid.

Closing here, as this appears to be triggered by flatcurve simply because it sets the FTS_BACKEND_FLAG_TOKENIZED_INPUT flag. But that's not a bug.

[doits]

Thanks for the investigations and detailed feedback!

Do you know how to contact the dovecot team about this since https://github.com/dovecot/core has issues disabled? 🤔

(Maybe you should leave this issue open so others with the same problem find it, even if it has to be fixed at dovecot core)

[doits]

Ah sorry, found it, I'll write a bug report by https://www.dovecot.org/bugreport-mail/

[doits]

Posted at https://dovecot.org/mailman3/hyperkitty/list/dovecot@dovecot.org/thread/XK4JVTAU3ECU25RKTSPLJ3BSVJJNEHN5/

[slusarz]

Ah, sorry, should have posted here: I already created an internal Bug ticket for this issue (internal ID DOP-3616). It's waiting for the team to triage.

[doits]

Any news about this? Is it maybe possible to view the internal issue for me, so I can follow it myself?