search

cypht-org / cypht

Cypht: Lightweight Open Source webmail aggregator [PHP, JS]. Supports IMAP/SMTP, JMAP and soon EWS
http://cypht.org
GNU Lesser General Public License v2.1
1.01k stars 164 forks source link

Mails missing from combined view (dovecot 2.3.21) #1195

Open indridieinarsson opened 3 months ago

indridieinarsson commented 3 months ago

🐛 Bugreport

I have cypht (2.2.0) running as docker container. I have 2 mail accounts, each on a different mail server. One of the mailservers is a self-hosted one (docker mailserver, running dovecot 2.3.21 as imap backend).

Both mail accounts work as expected when clicking the respective inbox in the "Email" drop down. However, all mails from the self hosted one (dovecot 2.3.21) are missing from all combined views (Everything, Unread , All etc.)

I have poked around a bit, and found the following search query being sent to the imap server SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED NOT HEADER X-Auto-Bcc cypht

I didn't quite go so far as to look into the actual reply from the imap server, but at least cypht parses it without any errors, and returns an empty array of messages (and I know there are some messages in this period of time).

If I just commented out some of the code that generates the search query, namely the following lines from hm-imap.php :

if ($only_auto_bcc) { $fld .= ' HEADER X-Auto-Bcc cypht'; } if ($exclude_auto_bcc && !mb_strstr($this->server, 'yahoo') && $this->server_supports_custom_headers()) { $fld .= ' NOT HEADER X-Auto-Bcc cypht'; }

making the search command look like this : SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED

After this change, the reply from the imap server is parsed, and the Everything view is populated with emails from both of my accounts.

indridieinarsson commented 3 months ago

Ok - I was a bit too quick to celebrate. When actually looking at the messages, I am getting all messages UNTIL 16-Aug-2024, but not the ones after that.
I must admit, that I don't know a lot (as in nothing) about imap standards and inner workings.

indridieinarsson commented 2 months ago

Getting messages from the wrong time period was just due to me messing with the search command (removed the UID part for some testing, forgot to add it back).

But it seems that my mailserver chokes on the NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ). It seems to get a positive answer from the server, the function returns true, but the subsequent search command returns no results.

marclaporte commented 2 months ago

Related MR: https://github.com/cypht-org/cypht/pull/1152 from @Shadow243

Shadow243 commented 2 months ago

NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ).

Just to confirm a doubt, can you please try to add the condition here: https://github.com/cypht-org/cypht/pull/1152/files#diff-2a16fbb7ff5f2c47ea84a2060f25affdc0fbeb9df48cff9f9438fa7ec705f624R1099

like

if ($only_auto_bcc && $this->server_supports_custom_headers()) {}
indridieinarsson commented 2 months ago

Thanks @Shadow243. I am already running with the code in the commit you mention, getting no messages from my dovecot mailserver if I run it as is. Only changes I have made is to comment out lines 1102:1104, then the search query returns the messages I expect it to.

I'm a bit uncertain what you mean exactly, could you perhaps specify more precisely what changes you want me to make.

Greetings, Indriði

indridieinarsson commented 2 months ago

I've looked into this some more, connecting to the server with pythons imaplib. The server is responding with OK and empty results list when searching with SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT HEADER X-Auto-Bcc cypht and also when searching with SEARCH (ALL) ALL SINCE "16-Aug-2024" HEADER X-Auto-Bcc cypht

If searching by custom headers wasn't supported, the server ought to answer with an error message stating that, I think.

This looks very much like the issue is with this (version of the) server, rather than cypht.

marclaporte commented 2 months ago

This looks very much like the issue is with this (version of the) server, rather than cypht.

@indridieinarsson Dovecot has a gigantic install base so this can affect a lot of people. I suggest to report the issue to Dovecot to see where they stand and what we can learn to make Cypht better.

indridieinarsson commented 2 months ago

I've been able to track this down to the dovecot-fts-xapian plugin for dovecot. The issue is not present if I disable full text search and use the default dovecot index instead (which is not nearly as flexible, and mostly unusable for real-life search in gigantic mailboxes), but re-appears as soon as I enable fts again.

In the source for dovecot-fts-xapian, I found that it is indexing only a selected (hard-coded) subset of email headers, and apparently returning an empty search result list for all searches that involve non-indexed headers.

Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

Shadow243 commented 2 months ago

Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

That's great. I spent a day looking for it in cypht, I couldn't even reproduce thought.

Can you provide more details so we can document it somewhere?

marclaporte commented 1 month ago

Re-opening because others seem to be getting this: https://github.com/cypht-org/cypht/issues/1259

So maybe we can detect the specific issue, and report an issue upstream? Then, maybe we can put a warning in Cypht?

indridieinarsson commented 1 month ago

Is it confirmed that #1259 is the same issue?

My issue (i.e. #1195) was due to a very specific peculiarity in fts-xapian: It indexes only a fixed set of headers, and returns an empty result set for all search queries involving other headers (i.e. "HEADER X-Auto-Bcc cypht" and "NOT HEADER X-Auto-Bcc cypht" are both empty) If so, there is a cypht-specific fix here : https://github.com/indridieinarsson/fts-xapian/commit/a4f0edc8fb5b1e646bdfa8f96409f52a87f263ae and here https://github.com/indridieinarsson/fts-xapian/commit/a0d52e747009780ef3582419e13b82ef9e52e796 Running cypht against a dovecot server with this version of fts-xapian is confirmed working.

marclaporte commented 1 month ago

Is it confirmed that #1259 is the same issue?

I don't know but symptoms are similar. One of Cypht's main selling points (compared to almost everything else) is email aggregation. So if there are cases where this fails, we should try to detect and inform the user how to solve.

Can you propose a PR to https://github.com/grosjo/fts-xapian? https://github.com/grosjo/fts-xapian/compare/master...indridieinarsson:fts-xapian:master

Thanks!