Make all IMAP searches use Xapian

[hagedose]

Discussion on the mailing list brought to light an issue that doesn't seem to be documented anywhere: it is my understanding that when search_engine is set to xapian, that index is currently only used for IMAP SEARCH FUZZY searches. Regular IMAP SEARCH commands apparently cause the server to actually search all messages (unless it's a header search). Prior versions of Cyrus did not support SEARCH FUZZY, but the SQUAT index was used for searches. At the very least it should be possible to specify both xapian and squat as search engines, so that both cases are covered. If at all possible I would argue xapian should be used for all searches.

[brong]

This was recently implemented on master:

https://github.com/cyrusimap/cyrus-imapd/commit/11f0bae309699d1b4e1eec7b7265af383d2cdb11

It should backport to 3.0 pretty easily

[rsto]

Thanks for reporting this.

We've already implemented this in https://github.com/cyrusimap/cyrus-imapd/commit/11f0bae309699d1b4e1eec7b7265af383d2cdb11

and it's going to be cherry-picked into 3.0

[rsto]

Ha, brong beat me to it :)

[elliefm]

The above-mentioned commit was already backported to the 3.0 branch (as e762c7cf9) on Wednesday. It'll be included in 3.0.8! :) (If you want it earlier, that commit should apply cleanly onto 3.0.7 sources, but ymmv!)

For clarity, this commit does not enable use of the Squat and Xapian engines at the same time, but rather, adds an option to treat all searches as FUZZY searches (and thus use Xapian for all searches)

It's documented in the imapd.conf(5) man page, but it'd be nice to clarify this detail and mention the setting it on the general Xapian instructions too -- will reopen and tag as documentation so it doesn't get forgotten!

[hagedose]

Thanks! I just built and installed version 3.0.7 with that additional commit, and the option works as expected. Unfortunately Xapian search itself always fails, but that is a separate issue ...

[dilyanpalauzov]

As reiterated at https://cyrus.topicbox.com/groups/info/T39b16104c6d0df13-M45c8814fde24a9265b56bc44/moving-to-xapian, the documentation for search_fuzzy_always does not state, that: • by default the Xapian index is not used for IMAP SEARCHES, but • when “RFC 6203 IMAP4 Extension for Fuzzy Search” is applied by the MUA the Xapian index is used.

The documentation shall describe also the downsides of enabling this option. E.g. the concurrence says for the Squat format:

The main difference between Squat indexes and the others is that Squat provides support for substring searches, while pretty much all other Full-Text-Search indexes support only matching from the beginning of words. By strictly reading the IMAP RFC it requires substring matching, so to optimize regular TEXT and BODY searches you must use Squat with Dovecot v2.0.