Message UIDs are not stable / possible data loss (deletion of wrong messages)

[richardweinberger]

My colleges and myself facing wired issues with ProtonMail Bridge and various mail clients (getmail, Kmail, Apple Mail). Symptoms include:

• Mail client re-downloads whole inbox, aborts, downloads, again, etc..
• Mail visible in ProtonMail web client but Mail client never sees or downloads it
• Altering some message A in webmail, but unrelated message B sees changes in mail client

All our mailboxes are large (50k to 100k mails) . This is maybe why we're facing the issue so easily.

Finally I found the proof that UIDs as presented by ProtonMail Bridge are not stable: As of now the bridge uses UID 51950 for the most recent message in my inbox.

a SELECT INBOX
* FLAGS (\Seen \SEEN \Flagged \FLAGGED \Deleted \DELETED \Draft \DRAFT $Junk Junk NonJunk)
* OK [PERMANENTFLAGS (\Seen \SEEN \Flagged \FLAGGED \Deleted \DELETED \Draft \DRAFT $Junk Junk NonJunk)] Flags permitted.
* 51950 EXISTS
* 0 RECENT
* OK [UIDNEXT 51952] Predicted next UID
* OK [UIDVALIDITY 4] UIDs valid
a OK [READ-WRITE] SELECT completed
a FETCH 51950 (FULL)
* 51950 FETCH (FLAGS (\Seen NonJunk) INTERNALDATE "29-Sep-2021 21:00:02 +0200" RFC822.SIZE 7129 ENVELOPE ("Wed, 29 Sep 2021 21:00:02 +0200" "Cron XXXX" (("(Cron Daemon)" NIL "root" "CENSORED")) (("(Cron Daemon)" NIL "root" "CENSORED")) NIL ((NIL NIL "root" "CENSORED")) NIL NIL NIL "<20210929191942.2F34CEBAD@CENSORED>") BODY ("text" "plain" ("charset" "utf-8") NIL NIL "quoted-printable" 4580 185))
a OK FETCH completed

As you can see UID 51950 is a mail from my cron daemon with date 29.09.2021 21:00. UIDVALIDITY is 4.

When I look at the getmail status file from my backup of 27.09.2021, I see the following line: 4/51950^@1632752979

The line decodes as "getmail downloaded mail with UID 51950 and UIDVALIDITY 4 on UNIX time 1632752979 (27.09.2021 - 16:29:39)"

This is the reason why I don't get the above mail delivered into my Maildir! getmail thinks it has already downloaded that message and skips it.

Expected Behavior

UID/UIDVALIDITY pairs should be stable such that mail clients have a chance to detect what mails are new.

Version Information

ProtonMail Bridge br-1.8.9 getmail 6.18.4

[m4rkw]

Apologies for somewhat diverting the point of the thread - I've removed most of my messages above and will summarise here.

1) I have specific examples of data loss - messages that were in my historical 14 days of local backups but which disappeared from my ProtonMail account after a 2-way sync operation. They are not visible in All Mail either. 2) I think prioritising a rewrite over fixing this issue was a mistake that could have put customer data at risk, particularly for anyone using the bridge with an IMAP client or for backups like I was 3) Point 2) would be forgivable if the issue had been communicated to users, AFAIK it still hasn't. There may well still be loads of people out there using UID-matching IMAP backup tools with no awareness of this problem, or worse IMAP clients with the bridge that occasionally delete messages and could end up deleting the wrong ones. This is really bad.

Additionally tangentially related, orphaned messages are a very counter-intuitive concept and should be eliminated from the system. I have around 17k emails that were not being backed up because of this. My suggestion would be to just associate all messages with Archive by default if they fall out of a folder. People using IMAP backup tools are very likely missing a lot of messages to this issue too just as I was.

[vmivanov]

@andrzejsza:

we have no confirmed examples of data loss

This is an outright false statement. I have mentioned twice in this bug report that I have lost emails. And while I cannot pinpoint the exact circumstances around when this happens, I see mismatched messages in Thunderbird on a regular basis, at least once every couple of days. As a result, a multi-select followed by delete cannot be guaranteed to have deleted the correct messages indicated by the metadata (sender, recipient, subject, etc) shown. The only reliable way to have "what you see is what you get" in a folder is to "repair" it every time which forces Thunderbird to re-fetch all messages. This is not a solution.

It happens so often on multiple machines, it's beyond me how a data loss scenario is impossible to reproduce by the Proton team. Simply disable "message synchronisation" on the account (which keeps copies of email locally) in Thunderbird (e.g. v91.9.1) and be patient. Before long there will be so many mismatched emails.

So I agree with @m4rkw [despite the unnecessarily long saga in this thread] above that users should be warned loud and clear that deletion using Bridge is not a safe operation - it can and will lead to loss of data. Now, "delete", by default on many clients will move the message to the designated Bin folder or the designated Archive folder, which is technically not "loss of data" but in the former case the bin is periodically expunged and then it becomes a data loss scenario.

I wonder how many users lost data thinking "oh no, I must have accidentally deleted it", when in fact it was Bridge that wiped their email(s). It's a natural reaction (we put too much trust in released software) and sometimes the mismatched email might date years ago which could explain why this is so under-reported as it might take many months or years for people to even realise if at all.

[m4rkw]

I've removed the saga and updated my summary comment above.

[m4rkw]

I wonder how many users lost data thinking "oh no, I must have accidentally deleted it", when in fact it was Bridge that wiped their email(s). It's a natural reaction (we put too much trust in released software) and sometimes the mismatched email might date years ago which could explain why this is so under-reported as it might take many months or years for people to even realise if at all.

100% this. Most people hardly ever look at emails older than today and when we do it’s usually a search. If I hadn’t been linked to this issue I’d likely never have realised my backups were incomplete and prone to missing data due to inconsistent UIDs. We can debate the merits of rewriting vs prioritising the fix but there’s no excuse for not notifying customers that you’re putting their data at risk.

[telephon]

I was just about to write: "Maybe the UID-mismatch is also connected to the issue of re-downloading?" (#29) and indeed it seems to be true, and probably fixed soon (see comment there).

[m4rkw]

The silence from Proton on this issue is deafening. I don't care anymore, have now migrated my email elsewhere.

[andrzejsza]

also done a bit of a cleanup to stick to the merit @bartbutler

@vmivanov

This is an outright false statement. I have mentioned twice in this bug report that I have lost emails. And while I cannot pinpoint the exact circumstances around when this happens, I see mismatched messages in Thunderbird on a regular basis, at least once every couple of days. As a result, a multi-select followed by delete cannot be guaranteed to have deleted the correct messages indicated by the metadata (sender, recipient, subject, etc) shown. The only reliable way to have "what you see is what you get" in a folder is to "repair" it every time which forces Thunderbird to re-fetch all messages. This is not a solution. Thunderbird mismatching messages is a long-standing issue of Thunderbird, that has also been reported on other, non-PM accounts. unstable UIDs are definitely contributing to the problem here on our side but it's very difficult to separate the two. That's why we're completely replacing go-imap with the new library.

@telephon definitely correct. The issue of resyncing has historically been the biggest challenge for us, before the problem of unstable UIDs was discovered - we are aware of some particular scenarios of resync not related to UIDs, which would also be fixed with the new library.

@m4rkw we appreciate the information you've provided and suggestions.

• we don't see how we can 'fix' the issue without the rewrite
• and yes, comms on this should have been better

[m4rkw]

• and yes, comms on this should have been better

Still should be better. Bridge users are probably mostly blissfully unaware that their messages might be getting lost but you don’t seem bothered about letting them know.

[jttttttttt]

A bunch of mails disappeared from my inbox in Thunderbird again, I had to uninstall Proton Bridge for the time being. I hope the new bridge will be ready soon.

[Amunak]

I'm equally surprised that there aren't more reports of this bug occurring.

Well I'm not; it's really hard to debug this kind of issues and I don't think many people suspect what we see as server-side application to be the issue.

I had issues with Thunderbird deleting/moving the wrong messages (I'd run a filter on the local mail then issue a delete/move command) and some of the messages that would get deleted/moved would be the wrong ones. I'm still not sure whether it has anything to do with this issue or Proton or if it's just a Thunderbird bug, but it's really annoying.

The saddest part is that all this could've been avoided if Protonmail - like any other sensible mail provider - had normal IMAP server on their side instead of the Bridge that's not even meant to run headless (so you effectively can't use, say, your own mobile email app).

[MorgothSauron]

I also encountered problem while using the bridge. I was doing some cleanup in some folders and I noticed I had the wrong emails were in the trash.

I don't know if its related, but way too often I have to repair a folder in thunderbird to see the right things. I either see duplicates or it simply doesn't show everything.

[JulienPalard]

and yes, comms on this should have been better

@andrzejsza still no plan to announce to bridge users that it can randomly delete their emails?

[karlemilnikka]

Yes, I’ve unfortunately had to start using the Bridge solely for backup purposes. It’s not stable enough for other usage.

[Tyler-2]

I have stopped using Bridge and my email client for anything involving search, older emails, deletions, or moves.

[Amunak]

Just FYI I think the issues have gotten better for me by clearing Bridge cache but again, not sure if it's even related.

[iodelma]

I wish that I had discovered these comments before because this issue with bridge cost me a job back in April. I have emails disappearing as well, then discover the occassional email in my inbox with a date that indicates it should have appeared sooner? This isn't bridge related, but there certainly are issues with protonmail and seriously considering another provider if someone can recommend one as I do not want to use gmail. Protonmail boasts security, but at what price, then this issue with bridge not being addressed after being brought to protonmail's attention, then where is the beta testing in the first place before releasing their products?

[MorgothSauron]

@iodelma Check at this list of private email provider: https://www.privacytools.io/privacy-email/

I personally used Startmail for a while and I was very happy with it. I stopped using it because they were dropping their business plan and there was no option to keep using this provider with my own domain. It seems that the use of custom domain is back and I'm seriously considering switching back to startmail.

The bridge reliability is really annoying. I stopped using any mail client to organize emails (move, delete). I even have problem with 'mbsync' that I use to synchronize emails locally. Since it uses the bridge, I face the same reliability issue. mbsync folders are often out of sync and I have to force a full sync. I shouldn't have to do that.

[iodelma]

Hello MorgothSauron,

Thank you very much for recommending Startmail - I really appreciate it!

Regarding bridge, this is completely unacceptable and I find it a bit shocking that protonmail has been aware of it for a year now.

Again, thanks for taking the time to offer me your suggestion, and so promptly as well. My annual fee is due soon, which is another issue I have with protonmail because they just charge my card without sending an invoice first, or at least a notification. It's just common practice, and I wrote to point this out to them. I received a reply indicating that they were looking into it as other customers had complained about this as well.

Sorry, off on a tangent, but now thanks to your suggestion, Startmail could be a solution.

Best, tony

[karlemilnikka]

@iodelma, I’m sorry to hear. I haven’t experienced any issues with the web version. All emails are coming through as they are supposed to. I am however using a custom domain (with correctly configured SPF, DKIM and DMARC). Are you using a custom domain as well?

(And yes, the Bridge is not close to production ready. I only use it to keep an offline copy of most of my emails.)

[iodelma]

Hello MorgothSauron,

Thank you very much for the link. A friend of mine suggested GMX for something free and secure. I'll certainly consider StartMail as well; I still have about two months before I pay my annual fee to protonmail, but I've decided to leave them considering how unstable it is, especially bridge. Yes, I have experienced the same issues and there are times I move an email to another folder and it disappears!

Anyway, without a commitment from protonmail to fix the problems with bridge, then aware of problems for a year, I'm leaving protonmail, so thanks again for the suggestions. It should be a concern that protonmail releases a product without sufficient beta testing. I don't consider this very secure from the user's point-of-view, but I guess if emails disappear, then no one else can see them either. I don't know what the issue is with their web version, though, in the end, it was a case of really bad luck that this important email was delayed? It was part of a message I sent via another email I received, which was stored in a separate folder and not in my inbox. As I noted, in Thunderbird, if I move emails to folders, there are times that they are deleted during the transfer. If they're important, then I have to resend to myself before moving them, or make copies. Anyway, time to move on, which is quite annoying as it means a new email address, etc.

[andrzejsza]

Again, please keep this thread for technical discussion about the issue - any valuable feedback is welcomed, especially regarding our new IMAP library. It's not a place for comments that are not related to message UID stability or even Bridge specifically. Not to undermine the problem here - a lot of client problems are not Bridge related. We need to look at almost all of the individually, so if you encounter any - please contact our support so that we can triage them, evaluate the logs and confirm the issue.

As for the merit - as you can see here: https://github.com/ProtonMail/gluon new IMAP library is actively worked on and we are in the process of integrating in within Bridge, changing underlying Bridge's architecture and improving test coverage. All of those activities are directly related to the issue reported here and are the team's top priority this year.

We will start alpha testing in the coming weeks with a target of Bridge V3 beta release by the end of this year. We are focusing on:

• IMAP correctness (with an assumption that any behaviour that goes against RFC3501 should be treated as a bug);
• stability and reliability;
• performance (thanks to both the new library and architectural changes we are seeing very significant improvements here). Above all, we are committed on testing.

[MqtgsN]

I have a similar but slightly different experience which it may be helpful to share. I have been watching this issue for quite a while after I finally figured out what was causing my woes, but I missed an incredibly important email the other day due to this problem, which has brought it back to the forefront of my mind.

Setup: I use mbsync to sync my mail from the Bridge, notmuch to tag it, and neomutt to browse it.

Emails get silently marked as read without being shown: I have thankfully not had any data loss (that I know of...), but I get the occasional email that never appears on my computer due to a previous UID seeming to be reused. The problem email then ends up marked as 'Read' on the server-side, so even when I log in to the web UI it doesn't stand out. It is only by comparing the list seen in the web UI with that seen in neomutt that can notice this.

Old emails surface in place of the correct one: I will sometimes, but inconsistently, see an old email (perhaps even from years ago) show up as unread. This generally tells me that the bug has surfaced and reused the UID that belonged to this old message. Something in my mail stack must have somehow detected that the 'read' flag has been changed on the mail, not quite sure which component. This is inconsistent though, so sometimes the bug just snipes a few of my emails silently.

Correlation to web UI activity?: I have a hunch that the occurences of this bug are correlated with my activity in the web UI. It seems to me that if I have recently logged into the web UI to do some management (edit folders, move mails, tag things, etc.), the bug is more likely to occur in the next day or two. If this is true, it unfortunately means that the user can get in a bit of a downward spiral. The more paranoid I am about the bug, the more I check the web UI, so the more the bug happens. However, I am not fully confident that my brain isn't just making up this pattern. I don't have time to spend all day trying to replicate this bug.

Attempt at limited mitigation with mbsync: I have now configured my mbsync to only ever 'Pull' and never 'Push' anything. Hopefully this means that the 'read' flag will not get pushed back to the server, meaning the affected mail won't get marked as read. This does effectively transform my email stack into an archiving solution only though, as I still need to check the title of all unread mails via the web UI to avoid missing anything.

Opinion: My understanding from comments in this thread is that the issue is non-trivial to fix without the coming re-write. Okay, that is understandable, but Proton should stop advertising this for the paid packages and communicate to all users that the Bridge is temporarily unstable and they should rely on the web UI for the time being. I understand that bugs occur and sometimes fixes can't be prioritised right away, but I think it is not okay to remain publicly silent about such a critical issue for over a year. As I mentioned in the beginning, I missed a really important email the other day due to this bug hiding it from me. Thankfully all was okay in the end as my contact prompted me via another channel that they had sent me this important mail, and knowing about this bug I immediately thought to search the web UI for it. Paying users who do not know about this bug will have been missing important emails for over a year now at least.

[MCKLtech]

+1 on this issue. I've seen it sporadically in the past ~4 weeks on MacOS Mail. The emails in MacOS Mail don't agree with those in the web browser. The web browser has the 'correct mails', which are just missing from the macOS Mail client.

This isn't a 'well take a look' kinda bug; this is a 'stop everything and fix it' bug. I can't believe this issue has been open for a year, Bridge recently got an update, and it's still happening.

At the very least, Bridge should be advertised as unstable and in beta as it's not up to the standard of a paid product. Proton is saying they are pleased with different platforms showing different emails; what sort of premium product is that? What are we paying for?

[iodelma]

Exactly.

I'm not paying again and turning to GMX. I've not read about any issues with it bridging to TB, which was the primary reason I've been paying protonmail.

Yes, this is a 'stop everything and fix it' bug and proton should advertise it as unstable, but then who would pay for an unstable service?

I've not bothered with the update as I'll be turning to GMX soon, but it's unacceptable that there is a release that hasn't resolved this problem.

Thanks for this feedback and hopefully proton will start listening.

On 01/11/2022 13:39, MCKLtech wrote:

+1 on this issue. I've seen it sporadically in the past ~4 weeks on MacOS Mail. The emails in MacOS Mail don't agree with those in the web browser. The web browser has the 'correct mails', which are just missing from the macOS Mail client.

This isn't a 'well take a look' kinda bug; this is a 'stop everything and fix it' bug. I can't believe this issue has been open for a year, Bridge recently got an update, and it's still happening.

At the very least, Bridge should be advertised as unstable and in beta as it's not up to the standard of a paid product. Proton is saying they are pleased with different platforms showing different emails; what sort of premium product is that? What are we paying for?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[cogabe]

+1

Same issue here, I have lost many emails and I have over 15k emails in total. Mostly some emails have the subject & senders from one email but the content is from another email. What is the beta channel for, when the production channel is full of bugs. Proton has been very unstable the last few months whether it's the mobile app or the bridge. I cannot wait for a fix, the damage has been done already.

People, leave or stay, but please have a backup, you are never too careful. Use the Proton Mail Import Export app.

[robertlagrant]

As for the merit - as you can see here: https://github.com/ProtonMail/gluon new IMAP library is actively worked on and we are in the process of integrating in within Bridge, changing underlying Bridge's architecture and improving test coverage. All of those activities are directly related to the issue reported here and are the team's top priority this year.

If there is an issue with the current Bridge, and this seems to imply that, then it's worth taking on the feedback that people don't feel ProtonMail are actually communicating this problem to their customers. That's why people are coming here to talk about it.

[oweidner]

Same here. Protonmail support staff advised me to use the "rebuild" function in Apple Mail. While this "fixes" the problem temporarily, things get quickly out of sync again.

[NeckBeardPrince]

https://www.reddit.com/r/ProtonMail/comments/yjz3yu/proton_bridge_message_uids_are_not_stable/

https://twitter.com/NeckBeardPrince/status/1587773487741018113?s=20&t=4pgzy4JUrOC38P-aoZmxFg

[vandenberghev]

I thought I was losing my mind, until i found this issue. Missing mails, old mails suddenly becoming unread, e.a. I stopped deleting e-mails in Outlook/Thunderbird and no more missing e-mails since.

Apart from having a seriously crippled experience right now: losing data is a big issue. I'm really disappointed in how this issue is treated. "Just wait for the new version" would not even be acceptable if there was a release date.

[ryukinix]

Mark the issue as wontfix, it's more honest

[andrzejsza]

no, we are working on it as per my last message here: https://github.com/ProtonMail/proton-bridge/issues/220#issuecomment-1254315208

[NeckBeardPrince]

no, we are working on it as per my last message here: #220 (comment)

You’re still not grasping what people are complaining about. We understand this is a bug and that you’re working to resolve it. But you NEED to communicate to everyone that this bug exists and inform users to its impact.

[dsommers]

And also push far more frequent updates (even if it is to development branches) so we can help testing. It's sad to see that only the end results are being pushed to the git repos, not the development process itself. As things is now, this is definitely far from an open source project. It's just an open sourced product for the time being.

[vandenberghev]

no, we are working on it as per my last message here: #220 (comment)

December 2021 you wrote:

the reason why we're not putting top prio on this at the moment is that we're doing a significant rewrite of Bridge's backed, including everyting that inovlves client communication. we have a strong level of certainty that this would resolve not only this, but also plenty of other issues.

Sorry if I misunderstood, but essentially what I interpreted is: "we've been working on a new version for over a year, which we hope will fix this" without even having reproduced the issue and/or having determined a root cause, and without any indication of an ETA for the new version. Even though I understand that this new version can (and probably will) fix the issue, you are, effectively, not working on this bug. So yeah, won't fix seems appropriate.

[dsommers]

@vandenberghev That's not how I read it at all. They are working on replacing go-imap with a new IMAP implementation. So lots of efforts has most likely been on that area. Once that work is ready to be consumed and implemented in Proton Mail Bridge, that's when changes starts happening in this repo.

Changing a core component (as the local IMAP server) is not an easy task, and that it has taken a year to do that is not surprisingly at all.

[NeckBeardPrince]

This thread is the exact reason why project managers need to be in the companies repos and issues. @andrzejsza we know you’re likely not the person with the power to announce to users of the bug to the real possibility (and probability) of data loss. So, who is? How do we reach the right person?

[MCKLtech]

@vandenberghev That's not how I read it at all. They are working on replacing go-imap with a new IMAP implementation. So lots of efforts has most likely been on that area. Once that work is ready to be consumed and implemented in Proton Mail Bridge, that's when changes starts happening in this repo.

If this were an open-source project with some people working in their spare time on it, I'd be perfectly happy with team efforts.

However, this is for-profit company, that charges for their services. I'm paying for this development work, or lack thereof. My own read of this situation is that the bug is much more involved that Proton are admitting to, and the fix is non-trivial, which is even more alarming.

Like I said above, this is a stop absolutely everything else and fix the situation. People are losing data and missing emails. At the very least, this issue should be communicated to all Bridge users so they can make an informed decision whether to continue to use it.

[jakw0j]

What are EU located alternatives to ProtonMail?

[dsommers]

What are EU located alternatives to ProtonMail?

Tutanota. But feature wise - including Proton Mail bugs - it is far behind Proton.

[ploum]

What are EU located alternatives to ProtonMail?

Mailfence (Belgium), Mailbox (Germany). None provide the same security as Protonmail (emails are, by default, not encrypted on their server). They provide a pretty good, robust and more classical service.

(Discovered this bug because some emails in my inbox are displayed are other email than on the webmail)

[dsommers]

@ploum I'd say those services you list may provide the same level of security, but not privacy.

[richardweinberger]

Guys, please stay on-topic. You can discuss this on https://news.ycombinator.com/item?id=33432296

[sharedphysics]

+1 to this issue and thought I was going crazy. Another (possibly related) issue I’ve noticed is that iOS mail with Bridge will show certain inbox-level emails within the protonmail bridge “all mail” folder. So I may receive an email, see it in my inbox at the top of the list in the mobile app or web app, but it is entirely hidden in iOS mail. Doesn’t show up in any inbox, doesn’t show up in unread email, but after poking around for long time I found it in the “all mail” view (for Bridge account, not even top level all accounts - which includes spam, filtered emails). Not a sync issue because this happens to unread emails in the inbox that can be days/weeks old and I’ll see newer emails come in.

[nicowilliams]

@richardweinberger wrote:

Well, what scares me more is that the bridge does not ensure stable UIDs. Which is a plain RFC violation.

RFC 3501 says:

A 32-bit value assigned to each message, which when used with the unique identifier validity value (see below) forms a 64-bit value that MUST NOT refer to any other message in the mailbox or any subsequent mailbox with the same name forever. Unique identifiers are assigned in a strictly ascending fashion in the mailbox; as each message is added to the mailbox it is assigned a higher UID than the message(s) which were added previously. Unlike message sequence numbers, unique identifiers are not necessarily contiguous.

The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions. Any change of unique identifiers between sessions MUST be detectable using the UIDVALIDITY mechanism discussed below. Persistent unique identifiers are required for a client to resynchronize its state from a previous session with the server (e.g., disconnected or offline access clients); this is discussed further in [IMAP-DISC].

In RFC 2119 parlance, SHOULD NOT is quite strong -- almost as strong as MUST NOT, and really means MUST NOT unless you have a really good reason to. In practice a lot of IMAP servers compute UIDs on the fly per-session with a new UIDVALIDITY, and then you lose because UIDs are not stable across sessions. This is very common for IMAP servers, so common that it's just best to never assume UID stability across sessions.

Typically IMAP servers use a timestamp or something as the UIDVALIDITY, and assign UIDs monotonically to all the emails in each mailbox that you open. Such IMAP servers have "stable" UIDs across sessions as long as you don't delete any emails, but the {UID, UIDVALIDITY} is ephemeral for each {session, mailbox} -- this isn't the stability that you're looking for!

@richardweinberger wrote:

I use getmail to fetch mails from IMAP into my Maildir. getmail has a simple text file where it stores the UID of each mail it has ever downloaded. Since 51950 was the UID of an mail at 27.09. it does not download new mail that has now UID 51950. Same applies to many other mails. As soon the bridge uses new UIDs getmail will download these messages.

Does getmail also store the UIDVALIDITY? If not, then that's not safe and a real and very serious bug. And because of the above, even if it did store both, the UID and the UIDVALIDITY for each email, it'd be useless anyways.

Instead IMAP clients have to SEARCH when they want to do anything with an email that they have cached locally. This is pretty much the only safe thing to do.

This is all why, for example, mutt re-lists the mailbox every time it reconnects to the IMAP server. Instead, mutt should just not care about UIDs except in the most ephemeral way, then the local header cache should be enough to avoid having to list the mailbox (you still want to do that to discover deletions by other clients, but that could be done asynchronously).

This is all very awful, but it's been a fact of IMAP life for a long time.

That said, there is no real reason that an IMAP server can't just assign every email in a mailbox a unique and stable 64-bit ID and split that between UID and UIDVALIDITY. It's just a bit more metadata to store and index by. It's not unreasonable to ask that any given IMAP server implement this, but it is real work to do this when it wasn't done that way from the beginning.

There's even no real reason that an IMAP server can't just do that across mailboxes since, after all, the RFC text Unique identifiers are assigned in a strictly ascending fashion in the mailbox; as each message is added to the mailbox it is assigned a higher UID than the message(s) which were added previously is not normative since it lacks RFC 2119 language! This, however, might be a bridge too far because there probably are clients that assume that UIDs are monotonically increasing and per-folder. But it'd be nice because you could move emails between folders during disconnected state and then sync that when next connected and it'd be easy. Gmail's label-based approach to folders was a game changer that IMAP can't easily keep up with on account of this UIDs-are-per-folder business.

All of this is even harder for IMAP<->whatever bridges, since if you want stable UIDs you'll need the bridge to keep state. But users and bridge devs want bridges to be stateless. Basically, a proxy that serves up IMAP can't be entirely stateless w/o help from the backend. If the backend is IMAP and has stable {UID, UIDVALIDITY}, or is not IMAP but has that feature anyways in order to help protocol transition proxies, then those can be stateless. But a general-purpose IMAP proxy/bridge really can't assume {UID, UIDVALIDITY} stability in the backend -- at best it can require it.

We need a time machine to fix the spec, and also the early implementations. Updating the spec can be done, but getting existing IMAP servers to implement it (and clients, too, where updates affect them) is like boiling a large lake.

[richardweinberger]

So you're implying that getmail, KMail, Thunderbird, mbsync and Apple Mail are the broken? Despite of the fact that Proton developers make clear in this thread that UIDs from the bridge are supposed to be stable?

[nicowilliams]

So you're implying that getmail, KMail, Thunderbird, mbsync and Apple Mail are the broken?

I don't know if they have such a bug or not. Idk if they store {UID} or {UID, UIDVALIDITY}. I'm not familiar with any of them.

Despite of the fact that Proton developers make cleat in this thread that UIDs from the bridge are supposed to be stable?

I guess I missed where they say that. ~Can you post a link to the comment where that claim is made?~ Ah.

But again, their notion of stability may not match yours.

We need to be very precise with terminology here. First, stability of UIDs alone is not a thing -- only {UID, UIDVALIDITY} assignments to actual emails can be stable, and only within a mailbox. Second, the stability period needs to be stated clearly: is it stable within a session only, or is it stable across all sessions (so, "forever")?

[nicowilliams]

As for the behavior of getmail described above, that is either broken or assumes stability guarantees that it should not.

[nicowilliams]

HN thread. Someone claims to have a fork to fix UID problems.

[richardweinberger]

TBH, I don't care anymore. I gave up on proton-bridge completely. My team faced problems with any client, not just getmail.

Hopefully proton-bridge developers can answer your questions.

[nicowilliams]

Eh, I care in an academic way. Best of luck.