Mails missing from combined view (dovecot 2.3.21)

[indridieinarsson]

🐛 Bugreport

I have cypht (2.2.0) running as docker container. I have 2 mail accounts, each on a different mail server. One of the mailservers is a self-hosted one (docker mailserver, running dovecot 2.3.21 as imap backend).

Both mail accounts work as expected when clicking the respective inbox in the "Email" drop down. However, all mails from the self hosted one (dovecot 2.3.21) are missing from all combined views (Everything, Unread , All etc.)

I have poked around a bit, and found the following search query being sent to the imap server SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED NOT HEADER X-Auto-Bcc cypht

I didn't quite go so far as to look into the actual reply from the imap server, but at least cypht parses it without any errors, and returns an empty array of messages (and I know there are some messages in this period of time).

If I just commented out some of the code that generates the search query, namely the following lines from hm-imap.php :

if ($only_auto_bcc) { $fld .= ' HEADER X-Auto-Bcc cypht'; } if ($exclude_auto_bcc && !mb_strstr($this->server, 'yahoo') && $this->server_supports_custom_headers()) { $fld .= ' NOT HEADER X-Auto-Bcc cypht'; }

making the search command look like this : SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED

After this change, the reply from the imap server is parsed, and the Everything view is populated with emails from both of my accounts.

[indridieinarsson]

Ok - I was a bit too quick to celebrate. When actually looking at the messages, I am getting all messages UNTIL 16-Aug-2024, but not the ones after that.
I must admit, that I don't know a lot (as in nothing) about imap standards and inner workings.

[indridieinarsson]

Getting messages from the wrong time period was just due to me messing with the search command (removed the UID part for some testing, forgot to add it back).

But it seems that my mailserver chokes on the NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ). It seems to get a positive answer from the server, the function returns true, but the subsequent search command returns no results.

[marclaporte]

Related MR: https://github.com/cypht-org/cypht/pull/1152 from @Shadow243

[Shadow243]

NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ).

Just to confirm a doubt, can you please try to add the condition here: https://github.com/cypht-org/cypht/pull/1152/files#diff-2a16fbb7ff5f2c47ea84a2060f25affdc0fbeb9df48cff9f9438fa7ec705f624R1099

like

if ($only_auto_bcc && $this->server_supports_custom_headers()) {}

[indridieinarsson]

Thanks @Shadow243. I am already running with the code in the commit you mention, getting no messages from my dovecot mailserver if I run it as is. Only changes I have made is to comment out lines 1102:1104, then the search query returns the messages I expect it to.

I'm a bit uncertain what you mean exactly, could you perhaps specify more precisely what changes you want me to make.

Greetings, Indriði

[indridieinarsson]

I've looked into this some more, connecting to the server with pythons imaplib. The server is responding with OK and empty results list when searching with SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT HEADER X-Auto-Bcc cypht and also when searching with SEARCH (ALL) ALL SINCE "16-Aug-2024" HEADER X-Auto-Bcc cypht

If searching by custom headers wasn't supported, the server ought to answer with an error message stating that, I think.

This looks very much like the issue is with this (version of the) server, rather than cypht.

[marclaporte]

This looks very much like the issue is with this (version of the) server, rather than cypht.

@indridieinarsson Dovecot has a gigantic install base so this can affect a lot of people. I suggest to report the issue to Dovecot to see where they stand and what we can learn to make Cypht better.