simple, single string body search exec'd as unneeded boolean OR?

[pgnd]

I've setup

dovecot-2.3.18-1.fc36.x86_64

and build/installed

dovecot23_fts_flatcurve-git.HEAD-0.20220520_110538.fc36.x86_64

After (re)scan + (re)index, if in Thunderbird I exec a search:

Search for messages in: pgnd@example
[X] Search subfolders
[X] Run search on server
[X] Match all of the following

[Body] [contains] [teststring]

with dovecot logging enabled as

log_debug = category=fts-flatcurve

I see in my logs, e.g.,

...
2022-05-22 20:11:36 imap(pgnd@example.com)<lNL/bKLf1t2sHgsH>: Debug: fts-flatcurve(some/path/subfolder): Query (body:teststr* OR body:teststring*) matches=0 uids=
...

That query,

body:teststr* OR body:teststring*

is an unneeded, basically redundant, boolean operation.

Bug, or config, problem?

[slusarz]

What are your Dovecot fts_filters settings?

It sort of looks like there is something weird going on with snowball filtering (or related). Since the first term is the stemmed version of the original query - without the English suffix ("ing"). https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_filters

[pgnd]

hi @slusarz

What are your Dovecot fts_filters settings?

currently, similar to the settings I'd been using with my prior fts_solr install,

...
mail_plugins = virtual acl fts fts_flatcurve
plugin {
    fts = flatcurve
    fts_enforced = yes

    fts_autoindex = yes
    fts_autoindex_max_recent_msgs = 999
    fts_autoindex_exclude  = \Junk
    fts_autoindex_exclude2 = \Trash

    fts_filters = normalizer-icu lowercase snowball stopwords
    fts_filters_en = lowercase snowball english-possessive stopwords

    fts_languages = en es de fr it pt
    fts_language_config = /usr/share/libexttextcat/fpdb.conf
    fts_tokenizers = generic email-address
    fts_tokenizer_generic = algorithm=simple

    # BUG: wait for dovecot 2.3.19 ...
    #  https://github.com/slusarz/dovecot-fts-flatcurve/issues/22
    # fts_header_excludes = *
    # fts_header_includes = From To Cc Bcc Subject Message-ID
}
...

[slusarz]

I can verify that this is happening, but this does not appear to be a flatcurve issue. It is, indeed, the snowball filter that causes two terms (the original term and the stemmed term) to be passed into flatcurve as an "OR" search.

Additional search terms I tested: "running" and "smashes". I see these queries:

May 23 22:51:29 imap(user)<139><ZS5ErrXftrJ/AAAB>: Debug: fts-flatcurve(imaptest): Query (body:running* OR body:run*) matches=0 uids=
May 23 22:51:29 imap(user)<139><ZS5ErrXftrJ/AAAB>: Debug: fts-flatcurve(imaptest): Query (body:smash* OR body:smashes*) matches=0 uids=

I will verify with the team whether this is the expected behavior of snowball.

Flatcurve has no way of knowing that the terms are related to each other. And it makes little sense for flatcurve to manually optimize these queries by filtering redundant search terms - that's quite complicated code, and Xapian index queries are plenty fast that this is not going to make any difference. So if this is truly the expected behavior of snowball, the current behavior is fine.

[pgnd]

@slusarz

fwiw, while troubleshooting tika searching, I exec'd a body search for a term "mairzy".

& noted in logs:

Debug: fts-flatcurve(INBOX): Query (body:mairzy* OR body:mairzi*) matches=0 uids=

notice that OR'd query is not just a redundant truncation, but an unwanted variant, namely:

mairzy*

, which would return desired results,

vs

mairzi*

, which would not

[slusarz]

So I think you answered your own question - the snowball filter does potentially provide independent terms that WILL result in different search results.

So maybe the snowball filter could be improved to not pass these terms to the FTS backend if they are redundant. But regardless, the snowball filter is not a flatcurve component, it's a core component, so this is the wrong place to be discussing that improvement. So I'm going to close this ticket (since, even with the redundant queries, flatcurve is returning the correct results).