format=flowed removed from Content-Type header

[jakecoble]

Messages sent with a Content-Type header set to text/plain; format=flowed arrive at their destination with their Content-Type header set to text/plain; charset=utf-8. The copy in ProtonMail's Sent folder also has this substitution.

Possible Solution

The bridge may be misunderstanding the encoding when format=flowed is present, reading the file encoding, and setting the Content-Type accordingly. I'm still investigating.

[exander77]

This is another case of ProtonMail messing up with email content and headers. This should not happen.

[jakecoble]

Just tested with Hydroxide and got the same result, so either they've got the same bug or ProtonMail's servers are mangling the Content-Type themselves. The second cause is far more discouraging than the first if true.

[exander77]

I can corraborate, this message:

Subject: Test Flowed 2
Content-Type: text/plain; charset="UTF-8"; format="flowed"

Flowed

Got turned into this message:

Subject: Test Flowed 2
Content-Type: multipart/mixed;boundary=---------------------06b44847e925faa623a7bd396c4cb76a
Date: Mon, 02 Nov 2020 19:18:23 +0000
Mime-Version: 1.0

-----------------------06b44847e925faa623a7bd396c4cb76a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset=utf-8

Flowed

-----------------------06b44847e925faa623a7bd396c4cb76a--

Not only it removed format=flowed. The plain message was turned into multipart.

This was done by server.

[jakecoble]

Also, sending it as multipart/mixed doesn't help—the server still removes format=flowed from the Content-Type.

[bartbutler]

This is true, the server translates the content type to a set of enums and format=flowed, if present, is then lost. This is unlikely to be reconsidered given the very minimal use cases and support of format=flowed in 2020, though I very much realize that not mangling format=flowed is a lower bar than supporting format=flowed in our clients. See the following blog post for something close to our current thinking on the subject:

https://fastmail.blog/2016/12/17/format-flowed/

[exander77]

@bartbutler I personally never heard about format=flowed, but I see this a symptom of a larges issue spelled out here: https://github.com/ProtonMail/proton-bridge/issues/120

This happens in body part headers as well. And such alterations are explicitly forbidden for the body part headers embedded in the bodies of messages of type "multipart" by RFC1341. Except for encryption purposes defined by different standards, ProtonMail should never alter any body part headers.

This in no way supporting something, this is about not violating standards.

[exander77]

Similar violation in both server and ProtonBridge is here: https://github.com/ProtonMail/proton-bridge/issues/99

[ploum]

For all those years (visionnary user here), I’ve been incriminating my mutt config. I never even thought that Protonmail could simply remove my format=flowed header. That’s really really bad.

[ploum]

. See the following blog post for something close to our current thinking on the subject:

https://fastmail.blog/2016/12/17/format-flowed/

Not supporting the RFC in the webmail is one decision. Modifying headers of emails sent by your customers is an entirely different matter and I consider it as an important violation of the trust contract between me, visionary user, and PM, mail provider selling trust.

[exander77]

. See the following blog post for something close to our current thinking on the subject: https://fastmail.blog/2016/12/17/format-flowed/

Not supporting the RFC in the webmail is one decision. Modifying headers of emails sent by your customers is an entirely different matter and I consider it as an important violation of the trust contract between me, visionary user, and PM, mail provider selling trust.

Yeah, I am thinking of leaving this service, altering e-mail content becoming unsustainable for me. So, you and me both.

[bartbutler]

We may reconsider how to support this properly, if not in the client editors but so that no mangling is done by Proton systems.

[ploum]

Regarding the support in the client editor, I think the fastmail blog can be summarized as : "people don’t use format=flowed because mail providers don’t support it and, as mail providers, we don’t use support it because people are not using it".

The political role of a mail provider is to implement what it believes is right and its weight will move the standard. Protonmail is putting is weight toward more PGP encryption (good) regardless of what is in use or not. Same could be said for any aspect of the mail standard.

Now, not willing to support the mail standard correctly for political reason (because PM believes some part of the standard are wrong) is a situation I accept as long as it is explicit and, as a customer, you have to make the choice of leaving or not.

[bartbutler]

In this case, it's not about format-flowed being "wrong". If support magically appeared tomorrow in our editors, I wouldn't be upset about it. It's about prioritization. It's probably not something we can justify devoting resources to supporting any time soon because relative to the other things we could do with those resources the upside/return-on-investment is just not very high. Not stripping it when it's already there is may be a different story because the effort required to do it may be lower (TBD).

[ploum]

I understand that. For me, the formal difference is between "we don’t want to support it" (aka fastmail), which is a political stance and "it’s in our bucket but not priority unless we perceive lot of demand for it" (which is technical stance).

I also believe that the fastmail post is confusing both stance and trying to justify the technical stance (making it looks harder than it looks) by coming with some weak political argument. Which is sometimes I find extremely irritating and hypocritical.

But I’m really happy to learn that PM position is evolving regarding this niche issue.

[exander77]

I understand that. For me, the formal difference is between "we don’t want to support it" (aka fastmail), which is a political stance and "it’s in our bucket but not priority unless we perceive lot of demand for it" (which is technical stance). I also believe that the fastmail post is confusing both stance and trying to justify the technical stance (making it looks harder than it looks) by coming with some weak political argument. Which is sometimes I find extremely irritating and hypocritical. But I’m really happy to learn that PM position is evolving regarding this niche issue.

Why are you bringing unrelated off-topic into a bug report? We are not talking about supporting format=flowed. We are talking about ProtonMail not messing with our e-mails and breaking them. What We think about format=flowed and what should mail providers implement is entirely besides the point. Mail providers should not modify e-mails and break them.

[husainaloos]

Any update on this issue?

[ploum]

I have the (maybe wrong) feeling that this issue is somewhat related to #180 . As #180 is announced as fixed (but not yet closed), is there any progress made on this issue ?

[exander77]

I have the (maybe wrong) feeling that this issue is somewhat related to #180 . As #180 is announced as fixed (but not yet closed), is there any progress made on this issue ?

It is all a singular issue. ProtonMail messes with e-mail content.

[Socob]

@exander77 Indeed. The discussions in your old issues, e. g. #18, #26, give some background on why issues like this one have existed from day one, and continue to persist for years.

[exander77]

@exander77 Indeed. The discussions in your old issues, e. g. #18, #26, give some background on why issues like this one have existed from day one, and continue to persist for years.

Yes, the issue is caused by parsing headers into header fields and some database structure (I assume) and throwing the original away, then trying to reconstruct it. So it obviously fails if it didn't know some extension like format=flowed was there, as it failed to parse it in the first place. It also has problems with tonne of non-standard, erroneous and even standard features, but corner cases. The solution is to parse everything as they need, but also preserve original headers, then serve those headers to the e-mail client. I don't think there is any other solution. MIME was not meant to be parsed and regenerated again.

Btw, I reported that "All Mail" should not contain "Spam" and "Trash" as it presents numerous issues (which are not really solvable - like moving e-mails from "All Mail" to "Trash" which cannot be really handled), they said tonne of things why not, but after 3 years, just a few months ago, they actually fixed it and set this behavior by default. So maybe they will fold in the end. I don't know.