search

mjl- / mox

modern full-featured open source secure mail server for low-maintenance self-hosted email
https://www.xmox.nl
MIT License
3.71k stars 113 forks source link

macOS's Mail.app IMAP folder matching #66

Closed x8x closed 1 year ago

x8x commented 1 year ago

When configuring a new account on macOS's Mail.app, the folder matching under Settings/Accounts/[select an account]/Mailbox Behaviours by default selects: Sent Mailbox to the folder Sent Messages (which is not on the IMAP server and so it gets created..) Bin Mailbox to folder Deleted Messages (same as above..)

This is confusing , there will then be a Folder Sent and Sent Messages and Trash and Deleted Messages. This can be see in the official screenshots: https://www.xmox.nl/screenshots/

As a user what I did was to go into settings mentioned above and match:

Somehow the fix (and we really need it!) resides on the server and it's communication to a maybe buggy macOS's Mail.app (Mail app on IOS devices doesn't have this problem).

Configuring a new account with a Profile which has no extra special settings with https://github.com/ProtonMail/proton-bridge/ does the proper folder matching when connecting to their server.

mjl- commented 1 year ago

I looked at an IMAP trace between Mail.app and mox just now. It looks like something is wrong with listing the mailboxes. Mox keeps track of "special-use" flags on mailboxes nowadays. But they don't seem to make it to Mail.app. Once that is fixed, I think Mail.app will use the Sent and Trash mailboxes created by mox for a new account. I'll dig in.

mjl- commented 1 year ago

The commit above fixes recognizing the Drafts mailbox, but doesn't fix the issue for macOS Mail.

Configuring a new account with a Profile which has no extra special settings with https://github.com/ProtonMail/proton-bridge/ does the proper folder matching when connecting to their server.

Is there any option for you to look at the IMAP commands between macOS Mail and the proton bridge? That might help understand why it is working with that combination, but not with mox.

Below is the IMAP protocol transcript with Thunderbird that I'm seeing for a new account. Note the "list" command that asks to return "special-use".

C: 58 authenticate CRAM-MD5
S: + [...]
C: [...]
S: 58 OK [CAPABILITY IMAP4rev2 IMAP4rev1 NABLE LITERAL+ IDLE SASL-IR BINARY UNSELECT UIDPLUS ESEARCH SEARCHRES MOVE UTF8=ONLY LIST-EXTENDED SPECIAL-USE LIST-STATUS AUTH=SCRAM-SHA-256 AUTH=SCRAM-SHA-1 AUTH=CRAM-MD5 ID APPENDLIMIT=9223372036854775807 CONDSTORE QRESYNC AUTH=PLAIN] authenticate done
C: 59 ID ("name" "Thunderbird" "version" "115.2.2")
S: * ID ("name" "mox" "version" "a0f3856e4005095378e6b1155945957272f9822c+modifications")
59 OK ID done
C: 60 ENABLE UTF8=ACCEPT
S: * ENABLED UTF8=ACCEPT
60 OK ENABLE done
C: 61 list (subscribed) "" "*" return (special-use)
S: * LIST (\Subscribed \Archive) "/" Archive
* LIST (\Subscribed \Drafts) "/" Drafts
* LIST (\Subscribed) "/" Inbox
* LIST (\Subscribed \Junk) "/" Junk
* LIST (\Subscribed \Sent) "/" Sent
* LIST (\Subscribed \Trash) "/" Trash
61 OK list done
C: 62 list (subscribed) "" "*" return (special-use)
S: * LIST (\Subscribed \Archive) "/" Archive
* LIST (\Subscribed \Drafts) "/" Drafts
* LIST (\Subscribed) "/" Inbox
* LIST (\Subscribed \Junk) "/" Junk
* LIST (\Subscribed \Sent) "/" Sent
* LIST (\Subscribed \Trash) "/" Trash
62 OK list done
C: 63 list "" "INBOX"
S: * LIST () "/" Inbox
63 OK list done

This is what iOS Mail (actually "iPhone Mail") does (with committed fix, on an account with different names for special-use mailboxes, trying to ensure email clients don't match on name):

S: * OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE LITERAL+ IDLE SASL-IR BINARY UNSELECT UIDPLUS ESEARCH SEARCHRES MOVE UTF8=ONLY LIST-EXTENDED SPECIAL-USE LIST-STATUS AUTH=SCRAM-SHA-256 AUTH=SCRAM-SHA-1 AUTH=CRAM-MD5 ID APPENDLIMIT=9223372036854775807 CONDSTORE QRESYNC AUTH=PLAIN] mox imap
C: 1 AUTHENTICATE CRAM-MD5
S: + [...]
C: [...]
S: 1 OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE LITERAL+ IDLE SASL-IR BINARY UNSELECT UIDPLUS ESEARCH SEARCHRES MOVE UTF8=ONLY LIST-EXTENDED SPECIAL-USE LIST-STATUS AUTH=SCRAM-SHA-256 AUTH=SCRAM-SHA-1 AUTH=CRAM-MD5 ID APPENDLIMIT=9223372036854775807 CONDSTORE QRESYNC AUTH=PLAIN] authenticate done
C: 2 ID ("name" "iPhone Mail" "version" "20G81" "os" "iOS" "os-version" "16.6.1 (20G81)")
S: * ID ("name" "mox" "version" "8c2814df89eb39d35b2b4d70b2b21bfe64a2b991+modifications")
2 OK ID done
C: 3 LIST "" "*" RETURN (SPECIAL-USE STATUS (UNSEEN))
S: * LIST () "/" Extra
* STATUS Extra (UNSEEN 0)
* LIST () "/" Inbox
* STATUS Inbox (UNSEEN 0)
* LIST () "/" Notes
* STATUS Notes (UNSEEN 0)
* LIST (\Drafts) "/" bezig
* STATUS bezig (UNSEEN 0)
* LIST (\Junk) "/" rotzooi
* STATUS rotzooi (UNSEEN 0)
* LIST (\Sent) "/" verstuurd
* STATUS verstuurd (UNSEEN 0)
* LIST (\Trash) "/" verwijderd
* STATUS verwijderd (UNSEEN 0)
* LIST (\Archive) "/" weggestopt
* STATUS weggestopt (UNSEEN 0)
3 OK LIST done

And macOS Mail (actually "Mac OS X Mail"):

S: * OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE LITERAL+ IDLE SASL-IR BINARY UNSELECT UIDPLUS ESEARCH SEARCHRES MOVE UTF8=ONLY LIST-EXTENDED SPECIAL-USE LIST-STATUS AUTH=SCRAM-SHA-256 AUTH=SCRAM-SHA-1 AUTH=CRAM-MD5 ID APPENDLIMIT=9223372036854775807 CONDSTORE QRESYNC AUTH=PLAIN] mox imap
C: 1.26 ID ("name" "Mac OS X Mail" "version" "16.0 (3696.120.41.1.4)" "os" "Mac OS X" "os-version" "12.6.9 (21G726)" "vendor" "Apple Inc.")
S: * ID ("name" "mox" "version" "8c2814df89eb39d35b2b4d70b2b21bfe64a2b991+modifications")
1.26 OK ID done
C: 2.26 AUTHENTICATE CRAM-MD5
S: + [...]
C: [...]
S: 2.26 OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE LITERAL+ IDLE SASL-IR BINARY UNSELECT UIDPLUS ESEARCH SEARCHRES MOVE UTF8=ONLY LIST-EXTENDED SPECIAL-USE LIST-STATUS AUTH=SCRAM-SHA-256 AUTH=SCRAM-SHA-1 AUTH=CRAM-MD5 ID APPENDLIMIT=9223372036854775807 CONDSTORE QRESYNC AUTH=PLAIN] authenticate done
C: 3.26 ENABLE QRESYNC
S: * ENABLED QRESYNC CONDSTORE
3.26 OK ENABLE done
C: 4.26 LIST "" ""
S: * LIST () "/" ""
4.26 OK LIST done
C: 5.26 SELECT INBOX (CONDSTORE)
S: * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $Forwarded $Junk $NotJunk $Phishing $MDNSent)
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $Forwarded $Junk $NotJunk $Phishing $MDNSent \*)] x
* 0 RECENT
* 0 EXISTS
* OK [UIDVALIDITY 847736430] x
* OK [UIDNEXT 1] x
* LIST () "/" Inbox
* OK [HIGHESTMODSEQ 1] x
5.26 OK [READ-WRITE] x
C: 6.26 LSUB "" *
S: * LSUB () "/" Extra
* LSUB () "/" Inbox
* LSUB () "/" Notes
* LSUB () "/" bezig
* LSUB () "/" rotzooi
* LSUB () "/" verstuurd
* LSUB () "/" verwijderd
* LSUB () "/" weggestopt
6.26 OK LSUB done
C: 7.26 IDLE
S: + waiting
C: DONE

Thunderbird and iPhone Mail use the extended list form, asking for special-use attributes: C: 3 LIST "" "*" RETURN (SPECIAL-USE STATUS (UNSEEN)) and C: 61 list (subscribed) "" "*" return (special-use). And they use them.

Mac OS X Mail is not using this "extended list" command. The list it executes is to find the mailbox-path-separator "/". It executes LSUB to list mailboxes, and LSUB is a deprecated command since IMAP4rev2.

I suspect that Mac OS X Mail is capable of asking for extended list (it also enables QRESYNC, which is more advanced), but that it decides not to. Perhaps it is looking for another capability in the greeting header, one that mox doesn't implement. I tried adding capabilities NAMESPACE (mox implements the command, it is included in IMAP4rev2, so not an extension anymore), CREATE-USE-SPECIAL (for making new mailboxes with special use flags, wasn't called), CHILDREN (mox probably also already implement it as part of IMAP4rev2). No change in behaviour by Mac OS X Mail.

A transcript with a request for special-use attributes by Mac OS X Mail will probably give a good hint for where to look.

x8x commented 1 year ago

macOS Mail.app having a conversation with Proton Bridge (sanitised):

* OK [CAPABILITY ID IDLE IMAP4rev1 MOVE STARTTLS UIDPLUS UNSELECT] Proton Mail Bridge 03.04.02 - gluon session ID 172
1.1 ID ("name" "Mac OS X Mail" "version" "16.0 (3731.700.6)" "os" "Mac OS X" "os-version" "13.5.2 (22G91)" "vendor" "Apple Inc.")
* ID ("name" "Proton Mail Bridge" "version" "03.04.02" "os" "darwin" "vendor" "TODO" "support-url" "TODO")
1.1 OK ID
2.1 LOGIN user@domain.com password
2.1 OK [CAPABILITY ID IDLE IMAP4rev1 MOVE STARTTLS UIDPLUS UNSELECT] Logged in
3.1 SELECT INBOX
* FLAGS (\Deleted \Flagged \Seen)
* 62 EXISTS
* 0 RECENT
* OK [PERMANENTFLAGS (\Deleted \Flagged \Seen)] Flags permitted
* OK [UIDNEXT 63] Predicted next UID
* OK [UIDVALIDITY 20269333] UIDs valid
3.1 OK [READ-WRITE] SELECT
4.1 FETCH 1:62 (FLAGS UID)
* 1 FETCH (FLAGS (\Seen) UID 1)
..
* 48 FETCH (FLAGS (\Seen) UID 48)
* 61 FETCH (FLAGS (\Flagged \Seen) UID 61)
* 62 FETCH (FLAGS (\Flagged \Seen) UID 62)
4.1 OK command completed in 5.609625ms
5.1 LSUB "" *
* OK [CAPABILITY ID IDLE IMAP4rev1 MOVE STARTTLS UIDPLUS UNSELECT] Proton Mail Bridge 03.04.02 - gluon session ID 173
1.2 ID ("name" "Mac OS X Mail" "version" "16.0 (3731.700.6)" "os" "Mac OS X" "os-version" "13.5.2 (22G91)" "vendor" "Apple Inc.")
* ID ("name" "Proton Mail Bridge" "version" "03.04.02" "os" "darwin" "vendor" "TODO" "support-url" "TODO")
1.2 OK ID
2.2 LOGIN user@domain.com password
* LSUB (\All \Noinferiors \Unmarked) "/" "All Mail"
* LSUB (\Flagged \Noinferiors \Unmarked) "/" "Starred"
* LSUB (\Drafts \Noinferiors \Unmarked) "/" "Drafts"
* LSUB (\Noinferiors \Trash \Unmarked) "/" "Trash"
* LSUB (\Archive \Noinferiors \Unmarked) "/" "Archive"
* LSUB (\Noinferiors \Sent \Unmarked) "/" "Sent"
* LSUB (\Noselect \Unmarked) "/" "Folders"
* LSUB (\Noselect \Unmarked) "/" "Labels"
* LSUB (\Noinferiors \Unmarked) "/" "INBOX"
* LSUB (\Junk \Noinferiors \Unmarked) "/" "Spam"
5.1 OK LSUB
6.1 STATUS "All Mail" (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
30.1 STATUS Starred (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
31.1 STATUS Archive (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
32.1 STATUS Drafts (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
* STATUS "All Mail" (MESSAGES 1876 UIDNEXT 1877 UIDVALIDITY 20269338 UNSEEN 0)
6.1 OK STATUS
33.1 STATUS Sent (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
34.1 STATUS Trash (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
35.1 STATUS Spam (MESSAGES UIDNEXT UIDVALIDITY UNSEEN)
* STATUS "Starred" (MESSAGES 5 UIDNEXT 6 UIDVALIDITY 20269329 UNSEEN 0)
30.1 OK STATUS
* STATUS "Archive" (MESSAGES 0 UIDNEXT 1 UIDVALIDITY 20269355 UNSEEN 0)
31.1 OK STATUS
* STATUS "Drafts" (MESSAGES 1 UIDNEXT 2 UIDVALIDITY 20269354 UNSEEN 0)
32.1 OK STATUS
* STATUS "Sent" (MESSAGES 18 UIDNEXT 19 UIDVALIDITY 20269334 UNSEEN 0)
33.1 OK STATUS
* STATUS "Trash" (MESSAGES 15 UIDNEXT 16 UIDVALIDITY 20269331 UNSEEN 0)
34.1 OK STATUS
* STATUS "Spam" (MESSAGES 2 UIDNEXT 3 UIDVALIDITY 20269345 UNSEEN 0)
35.1 OK STATUS
36.1 LIST "" *
* LIST (\Noinferiors \Trash \Unmarked) "/" "Trash"
* LIST (\Noinferiors \Unmarked) "/" "INBOX"
* LIST (\Archive \Noinferiors \Unmarked) "/" "Archive"
* LIST (\Noselect \Unmarked) "/" "Labels"
* LIST (\Junk \Noinferiors \Unmarked) "/" "Spam"
* LIST (\Flagged \Noinferiors \Unmarked) "/" "Starred"
* LIST (\Noinferiors \Sent \Unmarked) "/" "Sent"
* LIST (\Noselect \Unmarked) "/" "Folders"
* LIST (\All \Noinferiors \Unmarked) "/" "All Mail"
* LIST (\Drafts \Noinferiors \Unmarked) "/" "Drafts"
36.1 OK LIST
37.1 IDLE
+ Ready
DONE
37.1 OK IDLE
38.1 IDLE
+ Ready
2.2 OK [CAPABILITY ID IDLE IMAP4rev1 MOVE STARTTLS UIDPLUS UNSELECT] Logged in
3.2 LOGOUT
* BYE
3.2 OK LOGOUT
* OK [CAPABILITY ID IDLE IMAP4rev1 MOVE STARTTLS UIDPLUS UNSELECT] Proton Mail Bridge 03.04.02 - gluon session ID 174
1.3 ID ("name" "Mac OS X Mail" "version" "16.0 (3731.700.6)" "os" "Mac OS X" "os-version" "13.5.2 (22G91)" "vendor" "Apple Inc.")
* ID ("name" "Proton Mail Bridge" "version" "03.04.02" "os" "darwin" "vendor" "TODO" "support-url" "TODO")
1.3 OK ID
2.3 LOGIN user@domain.com password
2.3 OK [CAPABILITY ID IDLE IMAP4rev1 MOVE STARTTLS UIDPLUS UNSELECT] Logged in
3.3 SELECT Drafts
* FLAGS (\Deleted \Flagged \Seen)
* 1 EXISTS
* 0 RECENT
* OK [PERMANENTFLAGS (\Deleted \Flagged \Seen)] Flags permitted
* OK [UIDNEXT 2] Predicted next UID
* OK [UIDVALIDITY 20269354] UIDs valid
3.3 OK [READ-WRITE] SELECT
4.3 FETCH 1 (FLAGS UID)
* 1 FETCH (FLAGS (\Draft \Seen) UID 1)
4.3 OK command completed in 236.125µs
5.3 IDLE
+ Ready
DONE
5.3 OK IDLE
6.3 EXPUNGE
6.3 OK EXPUNGE
7.3 IDLE
+ Ready
DONE
DONE
38.1 OK IDLE
7.3 OK IDLE
8.3 UNSELECT
39.1 UNSELECT
39.1 OK UNSELECT
8.3 OK UNSELECT
9.3 SELECT Drafts
40.1 SELECT INBOX
* FLAGS (\Deleted \Flagged \Seen)
* 62 EXISTS
* 0 RECENT
* OK [PERMANENTFLAGS (\Deleted \Flagged \Seen)] Flags permitted
* OK [UIDNEXT 63] Predicted next UID
* OK [UIDVALIDITY 20269333] UIDs valid
40.1 OK [READ-WRITE] SELECT
* FLAGS (\Deleted \Flagged \Seen)
* 1 EXISTS
* 0 RECENT
* OK [PERMANENTFLAGS (\Deleted \Flagged \Seen)] Flags permitted
* OK [UIDNEXT 2] Predicted next UID
* OK [UIDVALIDITY 20269354] UIDs valid
9.3 OK [READ-WRITE] SELECT
x8x commented 1 year ago

After manual account setup led to a failed Mail.app connection, the Mailbox Behavior initially displayed Sent Messages and Deleted Messages. However, upon correcting the IMAP port to interface with Proton Bridge and achieving the first successful connection, the Mailbox Behavior automatically switched to display Sent and Trash.

mjl- commented 1 year ago

After manual account setup led to a failed Mail.app connection, the Mailbox Behavior initially displayed Sent Messages and Deleted Messages. However, upon correcting the IMAP port to interface with Proton Bridge and achieving the first successful connection, the Mailbox Behavior automatically switched to display Sent and Trash.

Thanks, that transcript was very helpful! It showed the special-use attributes were just sent without being requested. That is valid according to the RFC (rfc 6154), and Mac OS X mail used the flags in my testing, so it is what mox does now as well.

x8x commented 1 year ago

Thrilled that this is no longer a concern, thank you!