Two groups accidentally created while fetching pending messages after backup restore

[gerryfrancis]

• Operating System (Linux/Mac/Windows/iOS/Android): Android 11.

• Delta Chat Version: 1.44.0 (nightly built 2024-03-26).

• Expected behavior: When pending messages are fetched from the Inbox and Sent IMAP folders after a backup has been restored, emails that were sent by MUAs, e.g. Webmail clients, and contain Re: in the first place of the subject, e.g. in replies, do not trigger creating a new group named Re: [Subject].

• Actual behavior: When pending messages are fetched from the Inbox and Sent IMAP folders after a backup has been restored, emails that were sent by MUAs, e.g. Webmail clients, and contain Re: in the first place of the subject, e.g. in replies, trigger creating a new group named Re: [Subject].

• Steps to reproduce the problem: -- Create a backup of your Delta Chat account. -- Un- and reinstall Delta Chat, but do not launch the app, yet. -- Receive a classic email that has been addressed to you and a second recipient. (The email should remain in the Inbox folder of your IMAP email account.) -- Reply to that email by using a MUA client, e.g. Webmail. (Your email should be stored in the Sent folder with Re: [Subject] as the subject.) -- Launch Delta Chat and restore the backup of your account. (Result: Two groups are created, named Re: [Subject] (which contains the reply message) and [Subject] (which contains the first message), while pending messages are being fetched.)

• Screenshots:

• Logs: N/A.

[gerryfrancis]

FYI, it happens with version 1.45.0 (beta), too.

[iequidoo]

A fix i can imagine here is to sort by INTERNALDATE not only messages from a particular folder, but from all folders, before passing them to receive_imf_inner(). The problem is that folders are processed by different threads, so it will be actually not sorting, but some kind of synchronisation using e.g. tokio::sync::Notify + Mutex. Not too difficult to implement, but still quite a big change so it's a question whether fixing this is worth it. Maybe also a topo-sorting using the References header is a way.

[iequidoo]

Another possible fix is to leave this unordered processing of folders as is, but search Message-ID in the already known references, this way we can insert an unordered ("missed previously") message to the already existing chat.

EDIT: the problem is that several messages may be unordered, in this case we know the proper chat for them only when processing the last unordered message. Merging chats looks too complicated to fix this, some INTERNALDATE sorting-based solution could be better.

EDIT: A better possible fix: even if a message doesn't reference any known message, but has the same subject (without prefix) and "group members" as an already existing ad-hoc group, use that group for the new message.