Unicode tags not properly supported

[the-djmaze]

Steps to reproduce

1. Create tag/keyword/label in a different app (SnappyMail or Thunderbird)
2. Open NC Mail app

Expected behavior

See tags like "🧪" or "ກ ຂ ຄ"

Actual behavior

Vice-versa

1. Create tag/keyword/label in NC Mail app
2. Open in a different app

SnappyMail shows: Thunderbird can't handle them.

Mail app version

3.7.8

Details

IMAP 4.1 uses a modified version of UTF-7 IMAP 4.2 uses UTF-8

TAG3 SELECT "INBOX"
* FLAGS (\Answered … &2D7d6g-)

With NC Mail created tag

TAG3 SELECT "INBOX"
* FLAGS (\Answered … &2D7d6g- $&2d7d6g-)

NC Mail creates the invalid $&2d7d6g- and can't decode &2D7d6g-

[kesselb]

Thanks @the-djmaze for reaching out :+1:

I was just looking for another IMAP client that supports tags the other day. It's good to know that snappymail does.

Our handling of tags/flags in Nextcloud Mail is strange. The reason is, that horde/imap_client lowercases all flags by default and breaks the encoding. We are trying to be smart and uppercase everything before converting from utf7-imap to utf-8, but that doesn't work too well^1.

As a first step, we need to change, or add a new setter/getter, to read the unmodified flags: https://github.com/horde/Imap_Client/blob/98a4ba591bc4fa12341684d4051b344579a34e6f/lib/Horde/Imap/Client/Data/Fetch.php#L328-L331

I don't know if there's a reason that horde lowercases all flags by default.

cc @ChristophWurst @miaulalala

[miaulalala]

that's also kinda bad with registered IANA flags such as $Junk and $NotJunk - they're registered properties and should be treated as such by any mail client... related: https://github.com/nextcloud/mail/issues/10131

[the-djmaze]

that's also kinda bad with registered IANA flags such as $Junk and $NotJunk - they're registered properties and should be treated as such by any mail client... related: #10131

Yes, but what if there are tags in use $junk and $notjunk?

IMAP Formal Syntax says:

Except as noted otherwise, all alphabetic characters are case-insensitive. The use of upper or lower case characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion.

So $MDNSent, $mdnsENT, $MdNsEnT, $MdnSent are compliant.

Lowercase flags is ok, but should be done AFTER Horde_Imap_Client_Utf7imap::Utf7ImapToUtf8()

To prevent issues with servers that are case-sensitive (so wrong), setting a flag should be done with IANA registration $MDNSent

So @kesselb wants something cumbersome (when not fixing the other code) like:

    public function setFlags(array $flags)
    {
        $this->_data[Horde_Imap_Client::FETCH_FLAGS] = array_map(
            'Horde_Imap_Client_Utf7imap::Utf8ToUtf7Imap',
            array_unique(
                array_map(
                        'trim', 
                        array_map(
                            'Horde_Imap_Client_Utf7imap::Utf7ImapToUtf8',
                            $flags
                        )
                    )
                )
            )
        );
    }

Or when fixing the other code where Mail will always be UTF-8 (and only do Utf8ToUtf7Imap when sending command to server)

    public function setFlags(array $flags)
    {
        $this->_data[Horde_Imap_Client::FETCH_FLAGS] = array_unique(array_map(
             'trim', 
            $flags
        ));
    }