search

ProtonMail / proton-bridge

Proton Mail Bridge application
GNU General Public License v3.0
1.16k stars 157 forks source link

Receiving messages encrypted by 3rd party keys #28

Closed ettom closed 3 years ago

ettom commented 4 years ago

Let's say I recieve an email that is encrypted to my own PGP key. Currently, Bridge mangles the encrypted message and wraps it html with a message :

Decryption error
Decryption of this message's encrypted content failed.
openpgp: incorrect key

The PGP message that follows is mangled and cannot be decrypted. However, in the web client the same message is not mangled and can be decrypted with gpg -d. I find this behaviour very odd. Why is Bridge altering the content of the messages? Sending already encrypted emails fails too:

msmtp: the server did not accept the mail
msmtp: server message: 554 Error: transaction failed, blame it on the weather: backend: cannot create draft: cannot create attachment: Invalid input

I'd very much like to see this fixed. There should at least be an option to let incoming email pass through as-is.

Originally posted by @ettom in https://github.com/ProtonMail/proton-bridge/issues/4#issuecomment-620266008

jonathancross commented 4 years ago

Sending already encrypted emails fails too:

I can confirm this issue. I also saw it when using hydroxide suggesting that it is a server-side issue.

Fixing this is critical for those of us who wish to maintain full control of our PGP keys and / or use a hardware device such as a YubiKey, Nitrokey, etc. to encrypt email.

sagischwarz commented 4 years ago

It took my a while to find out that the problem with me not being able to decrypt received mails encrypted with my own PGP key was with the ProtonMail Bridge and not with my mail client configuration or PGP setup. I'd highly appreciate a toggle to disable all tampering with received and sent emails.

jonathancross commented 4 years ago

the problem ... was with the ProtonMail Bridge and not with my mail client configuration...

Note: it isn't even just Bridge per se, but the Protonmail servers which reject PGP/MIME encrypted emails that they don't control the keys to.

jonathancross commented 4 years ago

@cuthix Can you help us out here? Would be great to get a clearer understanding of what is going on (server side or in Bridge? etc.) and if it is being worked on. Thanks!

jameshoulahan commented 4 years ago

My understanding is that this issue (mangled armored PGP data in "decryption error" message) was fixed in v1.3.x of bridge; see the changelog, specifically issue number GODT-129.

Note that the issue regarding mangled armored PGP data is unrelated to the rejection of messages encrypted with keys we don't control. The scenario being discussed in this issue is:

As I believe the original issue was indeed correctly fixed, i will close this issue. If you can still reproduce the original issue on either v1.3.x or v1.4.x-beta of bridge, please reopen and i'll investigate what went wrong with the fix.

ettom commented 4 years ago

The original issue was not just about the PGP data getting mangled. It was about being able to use Bridge to send and receive emails encrypted by keys that Protonmail does not recognize, without the emails getting altered in any way.

The issue with mangled PGP data indeed seems to be fixed. However, Protonmail is still altering the content of emails. Incoming encrypted messages are wrapped in HTML and this breaks automatic decryption by 3rd party mail clients. Outgoing encrypted messages have their content type changed to multipart/mixed, contain the text "Empty Message" in both plaintext and HTML etc. While the original message is included and can be decrypted with gpg -d, I think this is still completely broken behaviour. The content of the mails should not be altered.

I think this should be reopened.

jonathancross commented 4 years ago

@jameshoulahan Where is the issue you mentioned RE: "rejection of messages encrypted with keys we don't control" if not here?

ettom commented 4 years ago

@jameshoulahan

If you can still reproduce the original issue on either v1.3.x or v1.4.x-beta of bridge, please reopen and i'll investigate what went wrong with the fix.

Github does not allow me to reopen the issue as it was not closed by me. Please reopen this.

jameshoulahan commented 4 years ago

Just confirming before I reopen: you are saying that v1.3.x or v1.4.x-beta of bridge still mangle the armored PGP data such that you are no longer able to manually decrypt it with gpg --decrypt? If so, I'll reopen the issue. Otherwise, I'd prefer to leave this issue closed, as the original bug has been fixed.

@jonathancross, I don't believe we have a github issue for that discussion yet.

ettom commented 4 years ago

As I said in my previous comment:

The issue with mangled PGP data indeed seems to be fixed.

...

The original issue was not just about the PGP data getting mangled. It was about being able to use Bridge to send and receive emails encrypted by keys that Protonmail does not recognize, without the emails getting altered in any way.

See my previous comment for a report of the current behaviour.

I don't believe we have a github issue for that discussion yet.

I believe this is the github issue for that discussion, as the problem of Protonmail rejecting the messages encrypted with keys they don't control was indeed brought up in the original issue. As @sagischwarz nicely put it: "I'd highly appreciate a toggle to disable all tampering with received and sent emails." Of course, if @jameshoulahan insists, I can open a new issue, but I fail to see how that would be productive.

jameshoulahan commented 4 years ago

I'll reopen the issue and see if I can loop in the product team.

Enabling integration with third-party PGP solutions is a source of ongoing internal discussion. I believe this is partially due to the impact it would have on other clients (web/android/iOS) and on the server. As such, please appreciate that we as bridge/ie developers don't have the final say here.

llzzrrdd commented 3 years ago

This is a major security problem with Protonmail! This is actually LESS secure than Gmail/Outlook... at least these services DO allow you to encrypt with your own keys. You OWN the encryption keys and do not allow your paying-for-extra-security sheep-customers to actually use the best practices out there? I am very disappointed and frustrated with such bug/limitation and as a customer of yours I do expect this fixed yesterday.

andrzejsza commented 3 years ago

@llzzrrdd we will almost certainly fail your expectation here. We don't consider this a bug but an expected behaviour - ProtonMail Bridge is designed to work with ProtonMail keys. That shouldn't be a surprise. We're closer to a final decision on how to handle messages encrypted with third-party keys and will definitely update here once we have a clear way forward.

seandlg commented 3 years ago

Gonna throw in my 2 cents here as well, as I believe that this problem spans beyond using PGP keys that Protonmail doesn't "hold" (obviously they're private to the user, but stored in encrypted form on the Protonmail servers).

As stated in the Protonmail Wiki, Email subjects are not encrypted, as this does not align with the PGP standard.

In contrast, many mail clients, probably most notably Thunderbird as of version 78, encrypt the Mail subject when using a built-in PGP feature (the Enigmail-stripped version does this automatically, currently without the option to opt out (2020/12/07)). For this reason, it often happens that I cannot use the web client to read PGP-encrypted messages sent to me by third parties.

This would be acceptable, if I could use my Mail client to read these mails. However, as stated per the issue, Protonmail wraps Mails it cannot decrypt into a layer of HTML, to display an error message to the user. This breaks compatibility with custom Mail clients, which could otherwise decrypt these mails. To the best of my understanding, the wrapping happens client-side (which makes sense, given @ettom's above statement that no HTML is added in the web client) here:

https://github.com/ProtonMail/proton-bridge/blob/a50266cdc086328a12976d1eaa0b33fc5488a4e8/pkg/message/utils.go#L64-L85

And the callee of interest is here (I believe):

https://github.com/ProtonMail/proton-bridge/blob/a50266cdc086328a12976d1eaa0b33fc5488a4e8/internal/imap/mailbox_message.go#L569-L574

It appears to me that a "Passthru-Option" would be a great feature for the Bridge. Deactivated by default, if enabled it would allow for receiving all kinds of PGP-encrypted mails and simply let the client handle decryption.

Notes

Given this context, is this a feature that you see room for?

bartbutler commented 3 years ago

@seandlg supporting encrypted subject decryption for the bridge should be fairly easy, and in any case, decryption for those messages should be fine--it's just a backwards-compatible extension to PGP/MIME, unless I'm missing something?

With regard to the pass through option, we should definitely do this. I think that the likely solution will be to include the decryption error message as a comment in the PGP ASCII armor.

seandlg commented 3 years ago

Yes, it is my understanding as well that subject encryption is a PGP/MIME extension. However, as per my understanding encrypted subjects are not on Protonmail's roadmap, partly because it renders Subject Searching in the Webapp impractical (see here). Nonetheless, the Passthru-Option would allow for working with these mails, using a custom mail client & the bridge.

dsommers commented 3 years ago

This issue does also extend to saving encrypted draft mails to an PM Bridge provided IMAP folder (like the PM Drafts folder). Nothing gets saved, as the Bridge rejects the saving, sometimes halfway after just adding a few headers. I see this regularly in Thunderbird 78.6.0 on RHEL 8.3.

Example of what gets saved:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=utf-8
Date: Sun, 03 Jan 2021 21:50:08 +0000
From: "John Doe" <john.doe@example.org>
Message-Id: <vG5kgzuER2dMNI7JxuWkaHJLIYYKEWoIqcbIbOalxQCd3wdBrAlK_KGMQOsCbS7IrseVGkpX4Vw_t4LlXzGcCQ==@protonmail.internalid>
Mime-Version: 1.0
References: <vG5kgzuER2dMNI7JxuWkaHJLIYYKEWoIqcbIbOalxQCd3wdBrAlK_KGMQOsCbS7IrseVGkpX4Vw_t4LlXzGcCQ==@protonmail.internalid>
Reply-To: "John Doe" <john.doe@example.org>
Subject: ...
To: "Jane Doe" <alice@example.com>
X-Pm-Content-Encryption: end-to-end
X-Pm-Conversationid-Id: 85e2zA_SSbP98Bg8xO_QDsP5zHGSm1_ab9KGWhdaCKPEoFKtIsT0sOVYUnWX64Uo1jxJVBOfqOD7ZTCnzVH15g==
X-Pm-Date: Sun, 03 Jan 2021 22:50:08 +0100
X-Pm-Internal-Id: vG5kgzuER2dMNI7JxuWkaHJLIYYKEWoIqcbIbOalxQCd3wdBrAlK_KGMQOsCbS7IrseVGkpX4Vw_t4LlXzGcCQ==
X-Pm-Origin: internal

Nothing of the encrypted data gets saved, essentially rendering "Save Drafts" useless if encryption is enabled.

While ProtonMail is an encrypted mail provider and service, there are times where even a higher security level is wanted and external encryption keys being important.

sneak commented 3 years ago

Bit by this again today. Simply preventing the HTML entities in the PGP ascii armor (such as &#43;) done by the templating engine would work to solve this.

This is super annoying - it means I basically can't receive PGP email (to my existing longstanding and well-known key) at my Protonmail address without jumping through a bunch of extra hoops.

dsommers commented 3 years ago

I've quickly tested this super hacky patch, which at least enables Thunderbird to decrypt messages with external PGP keys. It just removes the "Decryption error" message on the top of the PGP block.

This is just the first tiny step towards an "Advanced PGP mode" I've been advocating for several places. But more changes are needed, to avoid double-encrypting messages stored to in the ProtonMail account.

llzzrrdd commented 3 years ago

I've quickly tested this super hacky patch, which at least enables Thunderbird to decrypt messages with external PGP keys. It just removes the "Decryption error" message on the top of the PGP block.

This is just the first tiny step towards an "Advanced PGP mode" I've been advocating for several places. But more changes are needed, to avoid double-encrypting messages stored to in the ProtonMail account.

Cool! Now instead of "super hacky" can AT LAST ProtonMail officially provide such option in the bridge? We are all paying customers here and we are requesting this for so long...

dsommers commented 3 years ago

Cool! Now instead of "super hacky" can AT LAST ProtonMail officially provide such option in the bridge? We are all paying customers here and we are requesting this for so long...

As I said, this is just a very hacky and partial patch. It does not account for storing data in the ProtonMail account using external keys without ProtonMail ending up double-encrypting things and complicate this even further. It resolves only reading encrypted mails using external keys.

jameshoulahan commented 3 years ago

At long last, finally an update on this issue. A lot of work was put in recently to rewrite the message "builder" (the code that turns protonmail messages into RFC822 literals to be presented to email clients). The old builder was hard to work with and so we avoided making any significant changes to it. The new builder allowed us to finally tackle outstanding issues, such as properly handling externally encrypted messages.

With the 1.8.x release of bridge, messages you receive that were encrypted by an external sender should be handled correctly; if you use some third-party PGP solution (e.g. what thunderbird now has built-in), they should be decryptable and verifiable. This applies to both PGP/MIME and PGP/Inline messages (though PGP/Inline has the limitation that html bodies are presented as plaintext which might result in some nasty UX... that's the best we could do).

Please let us know if anything that should work still doesn't. Note that sending encrypted/signed messages (where you give bridge an encrypted/signed message to send, rather than letting bridge do the encrypting/signing for you) is still unsupported. Currently, if you wish to send an encrypted/signed message via bridge, you'll have to set up encrypting/signing within the settings of the contact you wish to send messages to, otherwise things might not work properly.

jonathancross commented 3 years ago

Thank you for the update. Great to see progress!

if you wish to send an encrypted/signed message via bridge, you'll have to set up encrypting/signing within the settings of the contact you wish to send messages

Does that mean that Bridge would need my private signing key and recipient's public encryption key?

Anyhow, the second (currently unsupported) use case is what is most important for me -- the ability to control my own keys (on hardware smartcard) and handle encryption / decryption outside of ProtonMail. Am I correct this is being actively worked on still?

llzzrrdd commented 3 years ago

I would fully agree with @jonathancross ...it is good to see some progress after all, but still the problem is there. Secure = let the users manage their keys if they want to. Why blocking this?

dsommers commented 3 years ago

I already left my appreciation of this update here: https://www.reddit.com/r/ProtonMail/comments/nltpfk/protonmail_bridge_182_thank_you_external_pgp_keys/

In addition to the encrypting/signing issue; PGP signatures on incoming clear text messages are also stripped out. The signature can be verified in the webmail client; but not in the IMAP client. What is even more puzzling me is how you achieve that; but that might be my ignorance to the PGP data format/spec. When I go into the webmail, select the text/plain message with a valid PGP signature and go to the "View headers" feature on the mail, I get this new embedded window with a the PGP encrypted body. Then decrypting that manually on my local computer with gpg I do see the plain-text message without a signature but with a gpg: section at the very end saying

gpg: Signature made Thu 27 May 2021 14:04:49 CEST
gpg:                using RSA key [.....]
gpg: Good signature from "[......]" [ultimate]

So somehow you've managed on the ProtonMail reception side to "embed" the signature in a correct way as part of the PGP encryption done on the plain-text message at reception. Even though that is neat and cool; it would be great to get that signature forwarded properly the IMAP client a swell.

And a final word for those needing the encrypting/signing issue when sending e-mail; I'm solving that now by using my own SMTP server what I have available for outgoing messages. Not ideal, but it works.

jonathancross commented 3 years ago

Thanks @dsommers ... So you are using your own SMTP server for outgoing messages, but with "reply to" your ProtonMail address? Does this cause red flags for Gmail recipients and the likes because the TLS certificate is not from protonmail.com?

Anyhow, I guess that is a bit off-topic, but it's good to know there is a straight forward workaround.

dsommers commented 3 years ago

Thanks @dsommers ... So you are using your own SMTP server for outgoing messages, but with "reply to" your ProtonMail address? Does this cause red flags for Gmail recipients and the likes because the TLS certificate is not from protonmail.com?

No, I'm using an SMTP server I fully control and manage, and my SPF record lists that server as an approved SMTP server for my domains. Oh, and I am also using my own domains; not any protonmail related domains.

That said; I saw that Bridge has another update today, which should also resolve "A bug with sending encrypted emails to external contacts"; it is unclear to me if that also accounts for external PGP keys or not.

llzzrrdd commented 3 years ago

"A bug with sending encrypted emails to external contacts" maybe @jameshoulahan knows something more?!

andrzejsza commented 3 years ago

Anyhow, the second (currently unsupported) use case is what is most important for me -- the ability to control my own keys (on hardware smartcard) and handle encryption / decryption outside of ProtonMail. Am I correct this is being actively worked on still?

no, this is not being actively worked on - we wanted to make sure that externally encrypted messages can be properly received. sending such messages is a completely separate problem for us due to technical limitations on the API and we are not looking into this.

just to make things clear - I've updated the title here as per original request - this is about receiving only.

Does that mean that Bridge would need my private signing key and recipient's public encryption key?

yes, Proton would need both.

@dsommers as for receiving plaintext signed messages - thanks for your comments and pointing this out, we will be fixing this in the near future (for internal reference GODT-1184)

A bug with sending encrypted emails to external contacts" maybe @jameshoulahan knows something more?!

nope, this is an unrelated issue - a hotfix for a bug introduced in 1.8.2.

andrzejsza commented 3 years ago

@dsommers in the recent beta, 1.8.5, we've fixed the problem you've mentioned - if message body contains an embedded signature, we are extracting the signature and building a multipart/signed message to preserve it.

dsommers commented 3 years ago

Thanks @andrzejsza has the git repo been updated? I'll spin a local build and test it out.

andrzejsza commented 3 years ago

yes, it is updated.

jameshoulahan commented 3 years ago

... but it doesn't work yet. I made a mistake -- the order of header entries is being modified which messes up the signature. It's been fixed, but @dsommers you'd be best to wait for 1.8.6 to hit this repo before you test it out! Apologies.

Alternatively, you can apply patch.txt to the top of master (it'll complain about whitespace errors because I removed some other stuff from the patch by hand + didn't care so much about lineendings when saving the file)

jameshoulahan commented 3 years ago

@dsommers Just letting you know, the repo has now been updated with a new release (v1.8.7) with all necessary fixes. I'd be keen to hear your feedback if signatures are preserved properly now.

dsommers commented 3 years ago

Just tested it.

I did all testing on git master commit 6be31bdeb2986f40c using a make build-nogui running the bridge using ./proton-bridge --cli.

jameshoulahan commented 3 years ago

Thanks for testing it!

Glad to hear receiving works as expected now. That was the aim of the changes I made.

Regarding sending, thanks for the feedback, but as mentioned above, sending encrypted/signed messages is still unsupported; as @andrzejsza put it, there are technical limitations on the API that need to be resolved first before we can make any more progress there.

dsommers commented 3 years ago

Fully understood on the sending part; that was primarily tested to satisfy the curious ones who might wonder if anything had changed :slightly_smiling_face:

andrzejsza commented 3 years ago

thanks for the summary @dsommers. let's keep this issue about receiving only - we'll be looking more into sending at some point in #26. since all works as expected now, closing.