Incorporate IMAP for the unified search results

[ChristophWurst]

Feature Request

The current unified search implementation (https://github.com/nextcloud/mail/pull/3433) does not search in bodies, as we just query the cached data in the database and we don't have the body there.

One of the technical limitations of cache-only search is that on IMAP we can't easily search across all mailboxes.

BUT there is a special IMAP mailbox attribute \All and we could use this mailbox (if it exists) to search in there. This won't give us a deep link to the message in the original mailbox, but at least we're able to find it.

As a fallback we could look into the INBOX, because there is a high chance people store most of their emails there, so we might find the message(s) the user is looking for there.

Summary

Use \All (or INBOX if \All is not available) mailbox to search email bodies on IMAP.

[dumblob]

The outlined solution sounds like a good idea. I'm all for it.

[StCyr]

Hello @ChristophWurst ,

I've had a look at this.

It seems that there is already some things related to IMAP search.

In particular there's this getIdsLocally function which seems to be function getIdsGlobally counterpart for IMAP search:

https://github.com/nextcloud/mail/blob/530d911fb5f0f5e2c040bdf0b9bdcdfbfbf9fecd/lib/Service/Search/MailSearch.php#L168

So, could it be possible that implementing the functionality be as easy as creating a new findMessagesLocally function, counterpart of findMessagesGlobally (which calls getIdsGlobally and is called by mail 'search provider 'search function) and updating mail 'search provider 'search function to also call this new findMessagesLocally function?!?

Maybe the IMAP search should be called as a background job though, as it may slow down mail 'search provider 'search function a lot?

[ChristophWurst]

Maybe the IMAP search should be called as a background job though, as it may slow down mail 'search provider 'search function a lot?

That's the original problem that we're facing with IMAP search. It's slow. We can't do anything in background because we need the results when we send them back to the client. There is nothing like partial or deferred results. Everything has to be ready at the same time.

So yes, once we add back search on IMAP, the search will be noticeably slower.

[ChristophWurst]

@StCyr IIRC we discussed https://github.com/nextcloud/mail/pull/3589 in the conf hackweek call. Our findings was that IMAP search is quite unreliable in performance as it depends on where the mail server is located and how fast it can answer a text search request.

So possible workarounds were discussed

• Only searching when the users explicitly adds body:xyz to their query -> limits usability to power users, plus you need to know where the info is
• Only search bodies for "fast" mail server, e.g. by guesstimating this based on IMAP capabilities
• (Possibly in conjunction with a point above) letting users configure whether search should include IMAP bodies or not via an account preference

[dodedodo]

I don't presume to have 'the' solution, just want to add some thoughts:

I think mail envelopes are too information-rich to properly show in the relatively small and simple unified search results box. For example; Date/time-stamps are already cut off with long titles, threading does not work (try searching "re:") and the upcoming tags feature probably won't fit either.

Existing email search (rainloop, roundcube, gmail, outlook etc) all have some kind of 'advanced search' button that reveals a UI to search specifically in from: to: subject: address: and the like. This way the user does not need to remember special search filter phrases.

Maybe mail could implement an 'advanced search' button that would show a modal pop-up, much like those other mail clients. It would show the results in the same envelope column you already expect all email to show. Another bonus is that there's more space to show the information-dense mail envelopes. By naming it 'Advanced Search' user shouldn't be confused by the 2 search options. Unified search can stay just the way it is right now (or maybe a bit more polished.. :shrug:).

Tapping into #4956, I think the padding proposed there would be a prime location for a search textbox. Very much like the rainloop UI. But I don't know how to sell that UX-wise, because there already is a unified search bar. Maybe it's acceptable if it shows a placeholder "advanced search"? But then again, where would the actual 'advanced search' go? Maybe it's a bad idea, but it would just fit so snugly there...

[dumblob]

@dodedodo I don't think any new modal pop-up is needed nor a good idea. Instead I'd prefer to go the "saved searches" route and reuse the existing Mail UI (imagine a list of historical searches instead of list of emails - clicking an item in it would open an advanced search form instead of an email).

Or if not history list, then just a list of saved searches while highlighting visually that the first item is special and is just an "advanced search" which will not get saved. This approach could be easier to implement and maintain as well as less disturbing to user experience (note there could or rather should be button(s) in the unified as you wrote which would directly jump to the advanced search form).

[dodedodo]

@dumblob Thx for your reply! I'm not suggesting showing results in a modal, just the advanced search form. This should hopefully make users a bit less confused by the 2 search options (ux). Where would you put an advanced search form?

If I understand you correctly, those 'saved searches' would act like permanent folders providing a new dynamicly generated 'filtered view', right? I like that idea. But I don't think this should be implemented using IMAP search. As discussed, the upside of imapsearch is to be able to search body text. But it demands a lot of time & resources on the imap server that would be reflected in the loading times for the saved-search/filtered-view. I also think it would feel odd UX wise to create a 'saved search', just for a one-off search.

[dumblob]

Because picture is worth thousand words I've spend almost 2 hours preparing a draft UI proposal (with 0 knowledge about Nextcloud design principles - so take it with grain of salt):

Does this make more sense to you @dodedodo ?

[nkinkade]

I would like to second @dodedodo on the notion of something like a modal advanced search box, and with results from the search displaying in the main mail messages area. The advanced search box could have a small warning that searching via IMAP on the server could be very slow. Here is what the Rainloop interface looks like:

I personally use Dovecot for IMAP and have Solr indexing my mail. My mail archive contains 80,000+ messages, and doing a full text search over every message doesn't take more than 2 or 3 seconds. Full text searching can be very fast, but you have to have your mail indexed in some way.

I've been using Rainloop in Nextcloud for a good number of years now. Not being able to search through the body of messages in Mail is one of the final things preventing me from migrating away from Rainloop. Rainloop is actually great, but the tighter integration of Mail with Nextcloud is appealing.

[nkinkade]

Am I wrong, or can this be closed as of v3.5.x with the introduction of doing message body searches via IMAP?