search

nextcloud / mail

💌 Mail app for Nextcloud
https://apps.nextcloud.com/apps/mail
GNU Affero General Public License v3.0
851 stars 263 forks source link

Instable connection to local hosted IMAP server #10199

Open Geryishere opened 2 months ago

Geryishere commented 2 months ago

Steps to reproduce

  1. Nextcloud is installed on own LXD container.
  2. I use XEAMS installed on another LXD container as local hosted IMAP server .
  3. Connection between Nextcloud Mail as client and Xeams as local host IMAP server is established without problems.
  4. SMTP Server connection is WEB.de SMTP server.
  5. After synchronising the email account on nextcloud mail with local IMAP, the IMAP server connection breaks down shortly after.
  6. In order to fetch the emails again, synchronising needs to be done manually (by reloading nextcloud mail).
  7. Then however, the IMAP connection breaks down again.

Expected behavior

The connection between Nextcloud Mail as email client and XEAMS used as local hosted IMAP server should be stable without interruptions.

Actual behavior

"reqId": "tXsefM4Jl5KSf6J64r6C", "level": 2, "time": "2024-09-28T12:07:41+02:00", "remoteAddr": "77.109.**.***", "user": "Gery", "app": "mail", "method": "POST", "url": "/apps/mail/api/mailboxes/31/sync", "message": "Mailbox UID validity changed. Wiping cache and performing full sync for 31", "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "version": "28.0.9.1", "data": { "app": "mail" }, "id": "66f7d5ce65aca"

[PHP] Error: Undefined array key "uidvalidity" at /var/www/nextcloud/apps/mail/vendor/bytestream/horde-imap-client/lib/Horde/Imap/Client/Base.php#4066 POST /apps/mail/api/mailboxes/95/sync from 77.109.**.*** by Gery at 28 Sept 2024, 11:32:53

"reqId": "pJdPemv2GN1DcQAjq51W", "level": 3, "time": "2024-09-28T11:32:53+02:00", "remoteAddr": "77.109.**.***", "user": "Gery", "app": "PHP", "method": "POST", "url": "/apps/mail/api/mailboxes/95/sync", "message": "Undefined array key \"uidvalidity\" at /var/www/nextcloud/apps/mail/vendor/bytestream/horde-imap-client/lib/Horde/Imap/Client/Base.php#4066", "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", "version": "29.0.7.1", "data": { "app": "PHP" }, "id": "66f7cdb1a0f34"

Mail app version

3.7.8

Mailserver or service

XEAMS hosted locally in the same network as nextcloud

Operating system

Ubuntu 22.04.5 LTS on LXD container

PHP engine version

PHP 8.3

Web server

Nginx

Database

PostgreSQL

Additional info

The same problem exists with Nextcloud Mail app in other Nextcloud versions (Nextcloud 27 and 28) running Apache2.

ChristophWurst commented 2 months ago

Mailbox UID validity changed.

This is a very rare event. We'll have to check if the code is still doing the right thing.

Geryishere commented 1 month ago

This event happens for ALL locally self hosted IMAP (mail)sever within the same network as Nextcloud. Only external (commercial) IMAP mailserver such as Google, GXM etc. are not affected.

kesselb commented 1 month ago

Could you please turn on the debug mode: https://github.com/nextcloud/mail/blob/main/doc/admin.md#logging

This will create a file "horde_imap.log" in your datadirectory with the imap traffic. If you spot the error in your logs again, then please post the related lines for the timeframe from horde_imap.log here.

Watch out for credentials or other information, the horde* logfiles are not redacted.

Geryishere commented 1 month ago

I had just installed mail version 3.7.9 before I got advice to create a file "horde_imap.log". I still experience the same behaviour as before: Undefined array key "uidvalidity". I will record error messages in horde_imap.log as suggested and then post it here.

Geryishere commented 1 month ago

I have generated the "horde_imap.log". This is the outcome:

Wed, 16 Oct 2024 20:13:10 +0000 Connection to: imap://192.168../ Server connection took 0.0002 seconds. S: OK Synametrics IMAP4rev1 server ready 10/16/24 10:13 PM C: 1 CAPABILITY S: CAPABILITY IMAP4rev1 IDLE S: 1 OK CAPABILITY completed Command 1 took 0.0003 seconds. C: 2 LOGIN **@local [PASSWORD] S: 2 OK LOGIN completed Command 2 took 0.0003 seconds. C: 3 CAPABILITY S: CAPABILITY IMAP4rev1 IDLE S: 3 OK CAPABILITY completed Command 3 took 0.0002 seconds. C: 4 EXAMINE INBOX S: FLAGS (\Seen \Deleted) S: 1 EXISTS S: OK [UIDNEXT 41] Predicted next UID S: 0 RECENT S: OK [PERMANENTFLAGS (\Deleted \Seen)] S: 4 OK [READ-ONLY] SELECT completed Command 4 took 0.0012 seconds. C: 5 UID SEARCH ALL S: SEARCH 40 S: 5 OK UID completed Command 5 took 0.0004 seconds. C: 6 UID FETCH 40 (UID) S: 1 FETCH (UID 40) S: 6 OK UID FETCH completed Command 6 took 0.0005 seconds. C: 7 UID FETCH 40 (ENVELOPE FLAGS INTERNALDATE BODY.PEEK[HEADER]) S: * 1 FETCH (UID 40 ENVELOPE ("Wed, 16 Oct 2024 22:10:36 +0200" "Testemail" (("=?utf-8?B?Ry4gSGVnZW5kw7ZyZmVy?=" NIL "**" "*.com")) (("=?utf-8?B?Ry4gSGVnZW5kw7ZyZmVy?=" NIL "**" "*.com")) (("=?utf-8?B?Ry4gSGVnZW5kw7ZyZmVy?=" NIL "***" "****.com")) ((NIL NIL "**" "*.com")) NIL NIL NIL "000001db2007$78e97fb0$6abc7f10$@**********@*****.com") FLAGS () INTERNALDATE "16-Oct-2024 22:12:25 +0200" BODY[HEADER] {8231} S: [LITERAL DATA: 8231 bytes] S: ) S: 7 OK UID FETCH completed Command 7 took 0.004 seconds.

Wed, 16 Oct 2024 20:13:10 +0000 Connection to: imap://192.168../ Server connection took 0.0003 seconds. S: OK Synametrics IMAP4rev1 server ready 10/16/24 10:13 PM C: 1 CAPABILITY S: CAPABILITY IMAP4rev1 IDLE S: 1 OK CAPABILITY completed Command 1 took 0.0003 seconds. C: 2 LOGIN ****@local [PASSWORD] S: 2 OK LOGIN completed Command 2 took 0.0003 seconds. C: 3 CAPABILITY S: CAPABILITY IMAP4rev1 IDLE S: 3 OK CAPABILITY completed Command 3 took 0.0002 seconds. CACHE: Using the OCA\Mail\Cache\Cache storage driver. C: 4 STATUS INBOX (MESSAGES RECENT UIDNEXT UIDVALIDITY UNSEEN) S: STATUS "INBOX" (MESSAGES 1 RECENT 0 UIDNEXT 41 UIDVALIDITY 10 UNSEEN 1) S: 4 OK STATUS COMPLETED Command 4 took 0.0006 seconds. C: 5 LOGOUT S: BYE IMAP4rev1 IDLE Server logging out S: 5 OK LOGOUT completed Command 5 took 0.0059 seconds. C: 8 LOGOUT S: * BYE IMAP4rev1 IDLE Server logging out S: 8 OK LOGOUT completed Command 8 took 0.0032 seconds.

Geryishere commented 1 month ago

Just to add the current error check in nextcloud.log:

"16 Oct 2024, 22:13:10" Warning mail
Mailbox UID validity changed. Wiping cache and performing full sync for 162 Error PHP Undefined array key "uidvalidity" at /var/www/nextcloud/apps/mail/vendor/bytestream/horde-imap-client/lib/Horde/Imap/Client/Base.php#4066

nextcloud-command commented 1 week ago

Hi there! It looks like this issue hasn’t seen much activity for a while and might need a bit more information to move forward. If we don’t hear back soon, we may close it to keep things organized. Thanks again for your contributions – we really appreciate it!

Geryishere commented 1 week ago

Unfortunately the issue is not solved although I have submitted all required information and never received a feedback. As I highlighted in my last comment, the problem is the use of horde-imap-client in nextcloud mail. It is NOT the compatible version which can used in Nextcloud as it interferes with PHP 8.2 and 8.3:

Undefined array key "uidvalidity" at /var/www/nextcloud/apps/mail/vendor/bytestream/horde-imap-client/lib/Horde/Imap/Client/Base.php#4066

SebastianKrupinski commented 1 week ago

Hi @Geryishere

I think this is more of a Xeams issue and not a mail app issue.

Let me explain, from the reading of your imap log, it looks like your mail server is NOT reporting a required value 'UIDVALIDITY' this is basically a sync token we use to determine if there have been changes in the mailbox.

Have a look at your examine response and mine.

Your mail server:

C: 4 EXAMINE INBOX

S: * FLAGS (\Seen \Deleted)
S: * 1 EXISTS
S: * OK [UIDNEXT 41] Predicted next UID
S: * 0 RECENT
S: * OK [PERMANENTFLAGS (\Deleted \Seen)]
S: 4 OK [READ-ONLY] SELECT completed

Form a different mail server:

C: 7 EXAMINE INBOX

S: * FLAGS (\Deleted \Seen \Answered \Draft \Flagged $MDNSent $Forwarded $AutoJunk $AutoNotJunk $Junk $NotJunk)
S: * 44 EXISTS
S: * 0 RECENT
S: * OK [UNSEEN 32] Message 32 is first unseen

S: * OK [UIDVALIDITY 1712247412] UID validity

S: * OK [UIDNEXT 179] Predicted next UID
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Answered \Draft \Flagged $MDNSent $Forwarded $AutoJunk $AutoNotJunk $Junk $NotJunk)] Permanent flags
S: 7 OK [READ-ONLY] SELECT completed

As you can see this line 'S: * OK [UIDVALIDITY 1712247412] UID validity' is missing in your server response.

Geryishere commented 1 week ago

Hi Sebastian, thank you for your detailed explanation. However, I am quite confused now as the answer given from XEAMS support on this issue says exactely the opposite:

Synametrics Support Date: 9/25/24 1:16 PM Topic: NextCloud + Xeams Type: General Discussions

"Gery,

A message that says "Undefined array key" appears to be a syntax error in one of the PHP files rather than a communication problem with IMAP server, and, therefore, is not related to Xeams."

Here is the link concerning this issue: https://www.xeams.com/app?operation=forum&st=viewOneArticle&id=16326

Is it possible that the problem is related to the Horde IMAP client used in Nextcloud Mail because "UIDVALIDITY" might not be caused by SEARCH but by EXAMINE? According to XEAMS support, most IMAP clients use the SELECT command, instead of EXAMINE command (see the details under https://www.xeams.com/app?operation=forum&st=viewOneArticle&id=16326).

SebastianKrupinski commented 1 week ago

Hi @Geryishere

The problem is Xeams, they are not implementing the IMAP RFC properly.

IMAP RFC - https://datatracker.ietf.org/doc/html/rfc9051#name-examine-command

The SELECT and EXAMINE commands are supposed to be identical, and the response returns are supposed to be the same. The only difference is that when EXAMINE is used, the mailbox (folder) should be opened in read-only mode. When SELECT is used the mailbox is opened and locked for write operations.

The error 'Undefined array key "uidvalidity"' is caused by the fact that UIDVALIDITY is a mandatory response variable, which is missing in the Xeams response. So Horde expects the UIDVALIDITY to be there, as its mandatory, but its missing which causes the error.

Sorry, but this is a improper implementation of the IMAP RFC on Xeams end and not Horde or Nextcloud.

Geryishere commented 1 week ago

Thank you for your sound arguments @SebastianKrupinski. Indeed the missing proper implementation of the IMAP RFC by Xeam needs to be reflected and I will share this issue with their support team. However how can it happen that other email clients than Horde or Nextcloud have no problems to share sync tokens with Xeams IMAP server. I tested MS Outlook and Google Android Mail App with Xeams IMAP and the exchange works perfectly. Even with Cypht Web Client, an IMAP communication can be established. Are these email clients functioning differenty from Horde/Nextcloud Mail?