[v3] Mail headers are rearranged

[dsommers]

When inspecting the e-mail headers, they are completely differently arranged compared to "View headers" via the Portal Mail webportal.

This breaks DKIM checks in mail clients, typically flagging them as suspicious. E-mail header ordering should be preserved via IMAP.

This behaviour can be tested in Thunderbird with the DKIM Verifier add-on.

Proton Mail Bridge version: git - v3 branch, commit 48480bc8391cee5b1797411e68d0821159f9b4fc Thunderbird version: 102.6.0 (64-bit) OS: Red Hat Enterprise Linux 8.7

Note: This is not a regression from prior Bridge generations, this happened there too. But I had hoped that the new gluon integration would resolve this issue.

[bartbutler]

Client-side DKIM checks are a lost cause for a number of reasons, it's probably not the header order, it's probably processing that happens when we convert to UTF8, encrypt, and sometimes do other manipulations.

[jameshoulahan]

While client-side DKIM checks might be somewhat out of our hands, we do indeed rearrange the headers as a side effect of go's random map ordering. This is a known issue and something that I hoped to sneak into v3 but never found the time (the message builder hasn't received much love for a while).

The API provides both raw headers (string) and pre-parsed headers (key-value map). We use the pre-parsed headers because the API tends to do a better parsing job than we do. However, the pre-parsed headers are unmarshaled into a go map, and as such, have undefined order.

[jameshoulahan]

I'm reopening the issue because maintaining deterministic message header ordering is something we should aim to do, regardless of whether or not DKIM checks will still work. There's one edge case I can think of where gluon actually relies on this.

I've opened a ticket internally to tackle this (GODT-2212).

[dikkechill]

I ran into this issue w.r.t. spam reporting to Spamcop. To determine the spam source, Spamcop specifically relies on the message header order. As the proton-bridge mail headers are not in the original order, I can no longer "automatically" submit messages with my mail client plugin.

The cumbersome workaround is to export the e-mail from the Proton web interface, copy paste the contents and submit messages manually. Next to helping shut down spam sources quickly, I can imagine there are other use cases that benefit from keeping the e-mail headers intact.

Would very much appreciate if this could be fixed. Thank you in advance!

[LBeernaertProton]

A fix for this issue was released with version 3.6.1. All new messages will have their header order preserved.

If you want to retroactively apply this, please remove the account from bridge and add it again.

[dikkechill]

A fix for this issue was released with version 3.6.1. All new messages will have their header order preserved.

If you want to retroactively apply this, please remove the account from bridge and add it again.

I'm using version 3.6.1 and don't see any change in behavior, neither for new or for existing messages (I removed and re-added the accounts). When I export the mail and check the headers from the web interface, they are still in a different order than when I check the message source in Thunderbird. E.g.

Web interface export (expected):

Return-Path: <service@admin.spamcop.net>
X-Original-To: <redacted>
Delivered-To: <redacted>
Received: from prod-sc-www03.spamcop.net (unknown [184.94.240.100]) by
 mailin042.protonmail.ch (Postfix) with SMTP id 4Sr1Pd4Yf4z3L for <redacted>;
 Wed, 13 Dec 2023 16:38:13 +0000 (UTC)
Received: from [redacted] by spamcop.net with HTTP; Wed, 13 Dec 2023 16:38:06 GMT
Authentication-Results: mail.protonmail.ch; dmarc=pass (p=none dis=none)
 header.from=cmds.spamcop.net
Authentication-Results: mail.protonmail.ch; spf=pass smtp.mailfrom=admin.spamcop.net
Authentication-Results: mail.protonmail.ch; arc=none smtp.remote-ip=184.94.240.100
Authentication-Results: mail.protonmail.ch; dkim=none
X-Spamcop-Conf: <redacted>
From: SpamCop robot <redacted@cmds.spamcop.net>
To: <removed>
..

Thunderbird (through bridge v3.6.1):

X-Pm-Gluon-Id: e57a1bc8-c04c-41a1-8ed4-ce87a8bdac81
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=utf-8
References: <HKS9ar9ri3zekH8QMkkxwYwlk8PCH17mLiRbYXohcPH2Y9MujW_1gYQw7In0ZRxAFmeaK0fhNDD4dZCpqQ9WUw==@protonmail.internalid>
X-Pm-Date: Wed, 13 Dec 2023 16:38:06 +0000
X-Pm-External-Id: <wh6579ddeeg2d37@msgid.spamcop.net>
X-Pm-Internal-Id: HKS9ar9ri3zekH8QMkkxwYwlk8PCH17mLiRbYXohcPH2Y9MujW_1gYQw7In0ZRxAFmeaK0fhNDD4dZCpqQ9WUw==
To: <removed>
Reply-To: "SpamCop robot" <redacted@cmds.spamcop.net>
From: "SpamCop robot" <redacted@cmds.spamcop.net>
Subject: [SpamCop] account configuration email
Mime-Version: 1.0
X-Pm-Spam-Action: inbox
X-Pm-Spamscore: 1
X-Pm-Content-Encryption: on-delivery
X-Pm-Transfer-Encryption: none
X-Pm-Origin: external
..

As you can see, even the Content-Type is somewhere in the top, where I would expect it to be at the start of the body.

[dsommers]

I haven't had time to do a thorough testing, but I had to setup everything again some days after the 3.7.1 (br-183) update. But my experience is quite similar to @dikkechill. It still feels somewhat re-ordered. Once I get a bit more time available, I'll run more thorough testing.