Mozilla Thunderbird - Retention Policies - Mail expired from normal folder still exists in 'All Mail' and in webmail 'All Mail'

[tangodev]

Expected Behavior

1. Thunderbird expires mail older than the threshold (ex. 1 day for notifications);
2. The same messages are also removed from all mail (and all mail in web);

Current Behavior

1. Thunderbird expires mail older than the threshold in notifications;
2. Messages expired in notifications are kept in all mail;
3. Those messages have the same symptoms as mentioned in this thread although the cause is evidently different (I did not remove folders);

Example: this message is in all mail but not anywhere else, note the missing folder icon.

Steps to Reproduce

Note: some of those steps may not be critical, but I'm including them for the sake of completeness.

1. Create a folder named notifications in ProtonMail Web;
2. Create a filter to move all emails from foo@example.com to notifications;
3. Install and configure Mozilla Thunderbird and ProtonMail Bridge (I'm not using "split addressing");
4. Wait for synchronization (in my setup, I synchronize [i.e. subscribe] to all folders, including all mail, spam etc.);
5. In Thunderbird go to rmb > properties on the notifications folder and add a custom expiry policy of 1 day;
6. Wait 2 days, make sure that you're actually getting mails from foo@example.com;
7. Check in Thunderbird and in ProtonMail Web the state of notifications and all mail;

Extra troubleshooting: manual delete

Note: requires setup as described in "Steps to reproduce".

1. In Thunderbird disable the expiry policy on notifications;
2. In Thunderbird manually delete the messages in notifications with delete;
3. In Thunderbird empty the trashcan;
4. In Thunderbird review the all mail folder: the message is deleted;
5. In Protonmail Web review the all mail folder: the message is deleted.

Results (expected and actual)

1. Messages manually removed from notifications get removed from all mail as well.

Version Information

Bridge 3.1.3
Kubuntu 22.04 LTS
Mozilla Thunderbird 102.10.0 (64-bit) 

Context (Environment)

The issue is making me less productive and making me hold on to emails longer than I want. I sort notifications and newsletters by how long I want to store them (ex. 1 day, 1 week etc.) and retention policies were supposed to enforce that expiry.

[LBeernaertProton]

Hey @tangodev. Thanks for the detailed report.

What I suspect is going on here is that Thunderbird is expunging the email from the local folder. Unfortunately, this does not cause the message to be "permadeleted" from Proton. Which is why it is still present in the All Mail mailbox.

We only "permadelete" messages when they are expunged from the Trash folder. You need to move the messages to Trash first and then delete them there.

[tangodev]

Hello @LBeernaertProton , thank you for the quick response!

I assume you're making the distinction between mark as deleted and expuge. Correct?

Yes, deleting from Trash works, but this is merely a workaround. Moreover, this is counter-intuitive, because I assumed:

Hey, it vanished, so it must be working!

This is bad from a UX perspective.

Is this something Bridge can account for, or is this just a no-op, because there's no reliable way to be sure that a policy was run and stuff should be expunged?

Regardless, I need an automatic workaround. Protonmail Web has no way to set expiry policies or to filter and move messages to the trash based on age. Mozilla Thunderbird filters based on age, but won't scope that per-folder.

ALSO: how can I safely clean up the messages that I assumed were expired? The ones visible in webmail all mail and without the folder icon? There's thousands of them and I don't want to screw something up.

[LBeernaertProton]

Due to a combination of bad IMAP clients and API limitation on Proton's side, we can't enable this behavior on all mailboxes at the moment.

I will raise subject with the Team, but I can't promise we will be able to address this.

[tangodev]

That's what I was afraid of, but I understand.

The best way to handle this IMHO would be to do retention policies server side. This bypasses the problem of Proton APIs (probably) and quirky clients, since Bridge (I assume) doesn't care for the reason why a message got deleted (purged) server side.

And the sales pitch for your Product Owner would additionally be: compliance. Companies small and large would benefit from some set it and forget it data rotation rules. Esp. when personal data is concerned. Even more so when people collaborate on the same data (shared mail) and the company wants things to be forgotten. Like a customer service inbox where people send all sorts of random junk.

To me this just makes sense for a company/service that wants to make data safer and attracts customers with the same mindset.

EDIT

You already seem to have the core functionality in place:

So the way I see it, the next step would be to expose a UI in the folder settings to allow users to set this per-folder.

[LBeernaertProton]

After discussion with colleagues, we can't support this due to API limitations that could lead to data loss.

What we would suggest is that you enable those settings in the web client and then add a rule in Thunderbird to move those messages to trash.

Finally, we can't add those settings to bridge as it is currently expected that all global settings for all clients be set via the web client.

[tangodev]

After discussion with colleagues, we can't support this due to API limitations that could lead to data loss.

Yeah, that's fair.

What we would suggest is that you enable those settings in the web client and then add a rule in Thunderbird to move those messages to trash.

Sadly, Tb doesn't allow to scope filters per-folder (unlike retention policies). But that's a me-problem.

Finally, we can't add those settings to bridge as it is currently expected that all global settings for all clients be set via the web client.

Makes perfect sense. As mentioned, the solution would be to add retention policies to folder option in the web client. You already do this for SPAM/Trash, so the underlying code seems to be there. It just needs to be adapted to this use case.

Are there plans to do this?

[LBeernaertProton]

Finally, we can't add those settings to bridge as it is currently expected that all global settings for all clients be set via the web client.