Feature: Add folder and label information into dedicated X-Pm-* headers

[dsommers]

When using ProtonMail Bridge with IMAP clients, the mail labels, folders and "All mails" folder can easily get messy. This is also due to lack of possibilities in the IMAP protocol.

Could we ask for PM Bridge to provide this kind of meta-data information in X-Pm-* headers? Like X-Pm-Labels: important, vendor X, customer Y and X-Pm-Saved-In: /Projects/Euphoria. These example values would be for an e-mail save in "Folders -> Projects -> Euphoria" with the the "important", "vendor X" and "customer Y" labels added to it. If looking at an e-mail inside the "All Mails" folder, the X-Pm-Saved-In header would still preserve /Projects/Euphoria.

The next step would be to extend IMAP clients to parse this information. For Thunderbird, this can be done via an add-on, which could perhaps reuse the "tags" feature it has. When looking in the "All Mails" folder, the "Location" column could be activated and the real folder the mail is saved in could be presented here.

Further, in Thunderbird, this could also either hide/remove the "Labels" folders - or make mails here "read only" - making it less confusing when deleting mails from a labels folder and the e-mail is deleted everywhere instead of just the label.

An integration this way would also make it a lot more convenient to use the pretty powerful search feature, where you can easily filter on tags as well saving these prepared searches. These searches can also span multiple e-mail accounts; going across e-mail providers.

[jameshoulahan]

Hi @dsommers, i think that's a really neat idea -- we'll discuss it internally and see if the rest of the team/company agrees. But just to be sure I understand: does the thunderbird add-on you describe already exist or are you proposing we develop such an add-on ourselves, to be used in conjunction with bridge?

[dsommers]

There exists a Thunderbird add-on for ProtonMail already, but it does not do much more than give some encryption status - which might be a reasonable starting point. https://addons.thunderbird.net/en-US/thunderbird/addon/protonmail-encryption-status/

I would like to help develop such an add-on, but I also know I have a pretty full plate at $PAID_WORK.

[polaris64]

I've just posted a feature request for a similar feature. My requirements are pretty similar, however I need this to work with mu which supports only X-Keywords, Keywords and X-Labels for tags. However if we got support for any type of custom headers then that would be good as I'm sure that I could map them to one of the ones that I need relatively easily at my end.

[cuthix]

Unfortunatelly the header of message must be preserved in IMAP. It means that if you move message from one mailbox to another we cannot change the header unless we assign a new UID into this messages (which client understands as delete and create messages).

On the other hand there is google extension: https://developers.google.com/gmail/imap/imap-extensions#access_to_gmail_labels_x-gm-labels

Which would be easier for clients (or extensions) to use and it will deliver the same information.

[dsommers]

@cuthix why do you think the headers must be preserved? For DKIM, everything below the DKMI signature must be unmodified. Otherwise it should not be an issue to extend with more data; which is what essentially happens whenever an e-mail passes an SMTP server, with the Received: headers.

[polaris64]

I've created a fork of ProtonMail Bridge so that I can use mu4e tags (stored in the X-Keywords header) to control the ProtonMail labels. The code is far from perfect and I plan to improve it, but currently it serves as a proof of concept that this can work. I have been using it for around a week now and there have been no issues at all.

This fork implements label/tag syncing for my own particular mail workflow (Emacs with mu4e and mu) where message tags are stored by mu4e in the X-Keywords header as a comma-separated list. However the idea could be adapted to work with any headers of course. Please see the amended README.md in the repository for details of how this works.

[eidetic-av]

Hi @polaris64, I have been testing your fork with mbsync.

Initially the X-Keywords are pulled from the server, but changes after the mail is initially synced do not propagate. Can I ask how you are watching for changes in X-Keywords and syncing them between your maildir client and the server? Are you using mbsync or offlineimap?

Thanks for your contribution. Perhaps the fork can be merged upstream after some further testing.

[polaris64]

Hi @eidetic-av,

Thanks for testing out my fork.

I am also using it with mbsync. There are two main problems at the moment: -

1. Simply changing keywords in a local message does not cause mbsync to push that message again to the remote server. So, if I want to change a message's tags I have to change them and then move the message to another folder, for example.
2. Similarly, changing the labels via Protonmail does not mark the message in any way as needing to be synced again, so again changes are only picked up if the message is also moved.

For problem 2 I was thinking about listening for events from Protonmail and then using that to mark the message as having been changed. For problem 1 there might be a way of making mbsync respond to modified headers, but I haven't looked into that yet.

Unfortunately I haven't had time to work on it since. I'd also like to merge the latest version of Protonmail Bridge (1.8.7) but there were quite a few upstream changes so again I haven't had time yet.

Despite these shortcomings I have been using the fork successfully for 3 months, so it would be great if we could fix the issues and eventually get this merged into upstream!

[eidetic-av]

Ok @polaris64 , I'm pretty clueless when it comes to this code-base right now, but if you could point me in the right direction I'd be willing to investigate further... (I understand if you don't have the time and I will continue anyway)

For point 2. I can see that labels are being updated in the store when they are changed on the server. For example, if I add or remove a label from the pm web app, the event_loop sees this and does in fact update the message here: https://github.com/ProtonMail/proton-bridge/blob/91dcb2f773f9c3c3286c653a2ec8eeee3d8f9a00/internal/store/event_loop.go#L569-L572

Although I'm not quite understanding yet why this update does not trigger a sync back to the server -- I suppose it only updates the running application's internal store?

Is the next step trying to hook into this event loop and then trigger your patched IMAP append?

[polaris64]

Hi @eidetic-av,

I'm also a bit rusty when it comes to the code-base, I haven't looked at it in any depth since my last commit!

My next step for sorting this issue was going to be checking exactly how mbsync identifies that a message needs to be pulled (i.e. go through a trace of the IMAP conversation) and then see if I can mark a message in a similar way whenever a message has had its labels changed in some way.

Listening to API events was one way I considered to detect the actual label changes, however this is not ideal as it requires ProtonMail Bridge to be running at the time of the change. That's also problematic if you run ProtonMail Bridge on multiple machines (as I do) as they're rarely running at the same time. Ideally this state would be identifiable from a future point in time (e.g. an API call to fetch all messages modified from the last sync), but this needs more investigation.

Just talking about it now though has got me interested in trying to figure this out again, so I'll try my best to do so if I can get some time :) Please let me know (either here or as an issue on the forked repository) if you have any luck yourself!

[eidetic-av]

@polaris64 Ok that makes sense.

There seem to be some working solutions to use imap with gmail tags (something like this), which seem to rely on headers in a similar way to pm. So maybe that is a place to start. My bad it's not based on IMAP.

I will keep you posted.

P.S. you have issues turned off (I think?) on your fork, which is why I was posting on this issue here.

[polaris64]

@eidetic-av, yes you were right, issues were not enabled on my fork. I've just enabled them now.

[polaris64]

@eidetic-av, I've been having a think about the second issue and I've been going through the code again to re-familiarise myself. I've had a few ideas about how this could be solved.

Firstly, I notice that message metadata is stored locally in a Bolt database. Therefore, upon fetching we could check if the message's LabelIDsfrom the API differ to those stored in the database. If they are different in any way then this is a sure sign that they keywords header value will now be different and that mbsync needs to fetch a fresh copy of the message.

The next step is a little trickier though. I tried indicating this change by setting a custom flag on the message, but it seems that mbsync does not support custom flags. I tried toggling the "\Seen" flag instead and that worked, but mbsync only syncs the flags in that case, not the body.

My next idea was to change the message's IMAP UID in that case, which should cause mbsync to treat it as a new message and download it afresh. However I don't really know enough about IMAP UIDs and UIDVALIDITY in order to do this properly. I'll try again once I've read more about it.

I've added this as an issue to my fork (https://github.com/polaris64/proton-bridge/issues/1), so please feel free to continue this discussion there.