The v3 version reverted the default message ordering

[blacklight]

Expected Behavior

When messages are downloaded on the new bridge (v3), they should be displayed in timestamp-descending order by default (that's how most of the clients render them).

Current Behavior

Messages are displayed in timestamp-ascending order (oldest first), breaking the default rendering on several clients. Tested on:

• K-9 for Android
• aerc
• mutt/neomutt
• Rainloop
• Nextcloud email

Thunderbird and Apple Mail can mitigate this issue because they support custom sorting on any column, but not all mail clients do.

Possible Solution

By default, messages should be downloaded and rendered in timestamp-descending order (newest first).

Steps to Reproduce

1. Download and build the new version of the bridge (v3)
2. Start the bridge
3. Refresh the messages on your client

Version Information

Latest checkout of the v3 branch

[blacklight]

Partial workaround: I have commented the reverse logic in internal/user/sync.go#127, removed the local store and forced a new sync.

But this definitely doesn't feel right - the order is fixed for the older messages, but some of the recent unread ones are still out of order (namely, at the bottom of the list).

Please, please, fix this stuff before the new release. I have been struggling with messages sorting issues on the bridge for months by now, I have implemented plenty of workarounds myself so far (and none of them is really satisfactory), and my experience with the bridge makes it barely usable. Please test the bridge against other clients as well before making a release - don't test it just against Thunderbird and Apple Mail and call it a day.

[gabor-meszaros]

Hi @BlackLight, Thank you for your interest in Bridge and bringing this topic up. The reverse logic you linked was implemented in order to help our users to use Bridge before the initial sync finishes. With that, the newest messages are shared with email clients early.

We indeed only tested this behaviour with Thunderbird, Apple Mail, and Outlook, but testing Bridge behaviour with real email clients is one of the most resource consuming part of our dev process; we have to make pragmatic calls on what we test with. Extending the current platforms and/or clients are not on the short term plan.

But I understand how annoying it can be with other clients. Once we are back in January, we will take a closer look if there is an easy solution (Internal reference: GODT-2231). If you can, please feel free to open a PR with your proposed solution. There we can discuss how it affects other clients / performance.

[blacklight]

Hi @gabor-meszaros, thanks for the response!

I understand that testing the bridge against several clients takes some effort, but I've already suggested in the past that testing against at least an email client other than the three you mentioned could really help prevent many of the inconsistencies with the IMAP implementation of the bridge.

The "problem" with Thunderbird, Apple Mail and Outlook is that, upon initial sync, they cache all metadata/message headers, and then they perform the sort internally according to some user-selected attribute (received date by default), regardless of the message UIDs or the order of the messages received by the server.

Many other clients (including those that I have previously mentioned), for performance or implementation reasons, don't synchronize the whole mailbox. Instead, they perform paginated IMAP requests. This means that they rely on the message UIDs that they receive from the bridge to sort things. (Note that K-9 for Android falls only partially into this category: it doesn't perform an initial full sync of the mailbox, it instead requires the user to scroll-to-update to download the next pages, but it sorts the message by date as new pages are received).

The v2 implementation had several issues with message UIDs ordering, which broke several of those clients in the past. The v3 implementation fixes those issues, but the reverse logic in the sync means that the clients will eventually consolidate the messages in an oldest-first order. This isn't a problem if the client downloads all the mailbox metadata and eventually sorts by received date, but it is a problem if the client keeps the messages in whatever order is received by the server, because the first page will contain the oldest messages.

I could open a PR with my proposed fix, but I 'm not sure if it breaks the expected behaviour on Outlook/Thunderbird/Apple upon initial sync. Again, I'd propose to test the bridge with one of the clients I mentioned (mutt/neomutt are quite well-known and documented, for instance), any of them should work because AFAIK they are all "uncached/unordered".

[blacklight]

Besides testing with other clients, many of the sorting issues on the bridge could be caught with a simple script like this:

from imapclient import IMAPClient

client = IMAPClient(host='localhost', port=1143, ssl=False)
client.login('user@email', 'password')
client.select_folder('INBOX', readonly=True)  # Or "All Mail"

# Get all the message IDs
ids = client.search(['ALL'])

# Fetch all the messages
data = client.fetch(ids, ['FLAGS'])

# Generate a message_id -> sequence number mapping for the messages
[(id, msg[b'SEQ']) for id, msg in data.items()]

Expectation: internal message IDs and sequence numbers should match. Actual behaviour: in my case, they do for many messages (after my changes that remove the reverse on sync), but some message IDs and sequence numbers still don't match. Without my changes, my hypothesis is that message IDs and SEQ numbers are in reversed order.

[jameshoulahan]

Hi @BlackLight, thanks for the detailed feedback. Please excuse the delay in replying, just got back from vacation.

Just to be sure we're on the same page: as far as I'm aware, there's nothing in RFC3501 that forces messages in a mailbox to be stored in "newest first" order. Indeed, the ordering of messages will, over time, get "messed up" as messages are moved between mailboxes. So what you're essentially asking for is a toggle in the bridge settings to control the initial sync direction?

You say many clients don't support custom reordering of messages. I wonder then what happens in the following scenario, which demonstrates how the message ordering can get "messed up":

Suppose you have mailboxes A and B. Mailbox A has three messages:	message	UID
A_a	1
A_b	2
A_c	3

Mailbox B has three messages:	message	UID
B_a	1
B_b	2
B_c	3

You then move all the messages from A to B, such that B has messages:	message	UID
B_a	1
B_b	2
B_c	3
A_a	4
A_b	5
A_c	6

You then move the A_c, A_b and A_a from B to A, in that order:	message	UID
A_c	4
A_b	5
A_a	6

Thus the mailbox ordering and message timestamp are completely separate things. Yes, adding a toggle in bridge to control sync direction might make it easier to work with clients that display messages sorted by UID, but the UID ordering might be completely different to the timestamp ordering.

[tristan957]

The mailbox ordering that I am seeing makes no sense. Is it by chance related to this issue. My email client is aerc, and Gmail messages sort just fine in my work account.

You can see 2021 messages sorted before some 2022 messages in the screenshot. I don't understand how to solve this since it isn't a problem with my client.

[LBeernaertProton]

@tristan957 Does your email client perform sorting by dates or does it just display the messages by UID or sequence ID?

[tristan957]

aerc will sort by date if the server supports it. If the server doesn't it's by UID

The bridge does not support sorting by date.

[tristan957]

@LBeernaertProton happy holidays! Hope you are having a great time with friends and family!

I am continuing to experience this issue on initial synchronization of an account through the bridge. I understand the Bridge is low priority for Proton given some huge percentage of users don't use anything but the web app and mobile apps, but those of us that participate in mailing lists for open-source development really rely on alternative email clients to get our jobs done, which means we have to go through the Bridge. When the Bridge has bugs like this, I have to use Gmail to do my contributions, and that just sucks because I hate Google, hence why I am a paying Proton customer.

Proton Mail in general just seems to lack features to support open-source development. No text-wrapping at a specified column width in the web app, no format=flowed (in the Bridge or the web app), and a poor Bridge experience. It has gotten to the point where I have considered moving to Fastmail. I wish Proton just had opt-in IMAP and SMTP endpoints, so we didn't have to host the Bridge ourselves and then we could use alternative mobile email clients.

Hell, I would probably pay a little bit extra per month if it meant we could have people solely dedicated to the bridge.

[LBeernaertProton]

@tristan957 Happy holidays and likewise.

I apologize for the lack of feedback. As much as we would love to implement all the requested features, we have to make though choices on where we focus our time.

We would like to treat everyone equally, but unfortunately, most of our users are either using Outlook, Apple Mail or Thunderbird. This in turns affects how we prioritize our work.

The only thing I can comment on at this point is that we have a ticket for this, but I can't guarantee any ETA.

Edit: Proton can't provide public facing IMAP server as that would require the servers to be able to decrypt the email messages, which is not possible and desired.

[tristan957]

@LBeernaertProton I can work around the issue if the following issue was resolved: https://github.com/ProtonMail/gluon/issues/378. Then I can tell my client to sort based on more parameters.

[blacklight]

@tristan957

I wish Proton just had opt-in IMAP and SMTP endpoints, so we didn't have to host the Bridge ourselves and then we could use alternative mobile email clients.

It's actually possible to achieve that through the bridge.

I have an instance of the bridge running on one of my servers, with an SSH tunnel that exposes its IMAP and SMTP ports to a VPN connection.

Any host on the same VPN can set the bridge's IP and ports as its IMAP/SMTP server, and (almost) everything works out of the box. I've even slapped Snappymail on it, so I have my custom webmail interface.

I wouldn't advertise this feature much if I were Proton, and I'd be careful setting it up myself, because a wrong bind address on the SSH tunnel and you're exposing a plaintext IMAP and SMTP interface to your mailbox to the world. But if you know what you are doing it's definitely worth giving it a try.

[blacklight]

Btw the sorting issue seems gone for me, but I guess it's just because I have a bridge that has been actively listening on my server 24/7 for the past year, so all messages are correctly received in a chronological order and UIDs are (almost) guaranteed to be monotonic.

My suspicion is that, if I were to start a new bridge today and sync, I may have the same issue again when using mutt, neomutt, aerc, or an alternative web client.

[tristan957]

Yes. This only occurs on initial sync. I don't host my own servers, so I don't think that the suggestion is viable for me, but hopefully it helps others.

[flogh]

Why don't you add a parameter to the login command for example, where we can choose if we want classic or inverted sync of the mail ? This would solve the issue. Not everybody is using Outlook or Apple mail.

I'm paying a service that is not working ..