Strange response from major website

[BobbyDavid69]

I am trying to set up a trade me account (a ebay like service for AU and NZ) but whenever sending me an activation email to my alias I get this very strange email in mandarin...

just really weird... get the same email with 4 different alias's now using different domains as well. Trade me is a very reputable website and this seems very strange for them to send out.

[nguyenkims]

It seems to be a an encryption problem, I wonder if you can try to create an account using an alias that has PGP disabled to see if the problem happens again.

Can you send me the URL to sign up for an account on this website so we can try to reproduce this issue as well?

[BobbyDavid69]

ok ill try disabling PGP and see what happens, strange its in mandarin though right? what would cause that?

of course, its www.trademe.co.nz

[BobbyDavid69]

ok well it is a PGP problem, I disabled it and now it works!

thanks for that! what would cause that issue though? I am interested as an IT student now :smile:

[salixh5]

Hey, I'm also having issues with text encoding and PGP. I'm also using ProtonMail, but not sure if that's the issue. For me, I was getting legitimate e-mails from Japanese government agencies etc. that weren't readable, so I tried out sending myself some mails with different text encodings. Seems like when the sender isn't using UTF-8 but some other encoding, the messages get garbled. In my case, the agencies were all using ISO-2022-JP encoding for sending mail.

Btw it looks like this when I send mail to myself from an external mail account (same content in each mail, only UTF-8 rendered correctly):

I've also checked to see if it works with PGP disabled, and yes, without PGP disabled I was able to read the mails just fine. So there is a solution, but it'd of course be great if I could leave PGP enabled somehow.

[salixh5]

For Japanese, you could test with this. However, I don't know exactly the issue from OP, but it's probably not really related to a specific language anyway.

A あ ア 亞

U+0041 LATIN CAPITAL LETTER A U+3042 HIRAGANA LETTER A U+30A2 KATAKANA LETTER A U+4E9E CJK UNIFIED IDEOGRAPH-4E9E

[FozzieHi]

Right, I think my previous testing was incorrect so this may be a universal problem. I'm not too experienced with encoding issues like this one so maybe @nguyenkims has more experience than me.

[salixh5]

Edit: I did all kinds of tests and could finally locate the problem somewhat. The issue is not with SimpleLogin and would happen completely in the same way with all other PGP implementations and mail servers.

The ProtonMail web-client is simply interpreting the message as UTF-8 when displaying it. If you export the message from ProtonMail directly without going through the web-client (for example, you can use the official Bridge client), then the message is completely correct, byte for byte.

My theory: When the mail is sent unencrypted, then ProtonMail servers will take the opportunity to re-encode the content to UTF-8 if needed. This means it can be displayed in a web browser without issues. For encrypted messages, it cannot do that. The web client is then not able to correctly display the message for that reason.

It'd maybe be a nice extra feature for the ProtonMail web client to be able to do this kind of conversion on-the-fly. OR an option for SimpleLogin to convert everything to UTF-8 before encryption would also be quite nice of course :)

[nguyenkims]

@jeifour thanks for the investigation! SimpleLogin simply encrypts the email content (in bytes) and doesn't do any decoding in most cases.

[salixh5]

This is an example of an encrypted email I got in my ProtonMail inbox. The only thing they (can) do is change the Version string, the rest is verbatim like it was sent by SimpleLogin: https://gist.github.com/jeifour/66975337580e66bc21cdf71a8b07da7f

I can't give out my private key of course but here's this exact email after decryption: https://gist.github.com/jeifour/43ca306eb3d46d1f0187e4f238397842

It's totally valid and when you save it as .eml and open it up in Apple Mail, Thunderbird or some other email client everything is displayed correctly. However, even though the ProtonMail web client can decrypt it fine, it cannot display it correctly, with the encoding being the likely issue here.

Apple Mail:

ProtonMail web client:

Maybe I should just start using the ProtonMail Bridge to read my mail. Don't see another solution for this issue (other than disabling PGP in SimpleLogin altogether). But for good measure, I tried reaching out to ProtonMail via their support. Not sure if I'll be able to describe the issue correctly so that they understand, as English isn't my first language, but it's worth a try.

[nguyenkims]

@jeifour I wonder if we should open an issue on https://github.com/ProtonMail/WebClients so Protonmail team is aware of the issue?

[salixh5]

@nguyenkims I should be getting an update on this from ProtonMail support soon. I'll post about it in this issue. In the mean time, if anyone wants to reproduce this for themselves without using SimpleLogin at all but isn't sure how exactly, this was my setup (using macOS, but should work on other operating systems):

1) Install SeaMonkey with Enigmail add-on. As easy as brew install seamonkey and then installing the latest Enigmail version for Thunderbird from their website.

I used SeaMonkey because it's still getting security updates and works with modern mail servers, and it fully supports sending e-mails with a text encoding of your choice. Before that I tried several other mail clients (Outlook, Thunderbird and more) but had no luck. Maybe there's a better alternative for Linux and/or Windows.

2) Set up a non-ProtonMail mailbox in SeaMonkey. I used my university's mail servers and they worked fine, but probably anything like Gmail or Yahoo should work equally well for this test.

3) Import your ProtonMail addresses public key into your GPG keyring. You can easily get the public key from any ProtonMail user through the URL https://api.protonmail.ch/pks/lookup?op=get&search=username@protonmail.com

4) Now compose a new e-mail in SeaMonkey, change the Text Encoding from the menu and use some characters in your test message that are specific to that encoding. For example, you could use ISO-2022-JP and write Aあア亜. Make sure the SeaMonkey composer window mentions your text encoding in brackets in the title, that means it is actually applied. If you try to use characters that don't work in your selected text encoding, it might automatically switch back to UTF-8.

5) Lastly, click on the Enigmail Encrypt button in the composer window so that your email gets encrypted before sending. Then send away to the ProtonMail e-mail address!

Result: Even though SimpleLogin wasn't involved here (direct e-mail from a non-ProtonMail mailbox to a ProtonMail mailbox with nothing in between) the ProtonMail web client will have display issues. It will work fine however if you read your ProtonMail e-mails through their Bridge. I think therefore this is a general issue with how their client displays emails.

As mentioned above, I'm currently still waiting for ProtonMail support's response to this, but it seems like in general this would be a good fit for a GitHub issue on their side. However, I'm also having the same issue on their iOS client that is currently publicly available in the App Store. It would be interesting to see how these messages are displayed in their Android client and their new beta iOS client. I'm not a part of beta testing for that new iOS client, but maybe it's fixed there.

[salixh5]

ProtonMail stated that they can reproduce some decoding issues in their iOS app. However, it's still apparently not clear what exactly triggers the problem in the web client. I've sent them some additional information so hopefully it can get reproduced successfully.

For some reason, when I send e-mails with Shift_JIS encoding for testing now they display fine in the web client with only the iOS app not displaying them correctly. Other encoding still seem to be affected by the issue, though. This kind of inconsistency makes me doubt my earlier assumptions being 100% true. If I have the time, I'll try to assess the situation more systematically instead of doing random tests.

[salixh5]

I poured some time into testing this, but ultimately I feel like I understand less about the root cause than before. Maybe someone else can give a hint here or has an idea. Here is what happened.

Direct tests

To establish a baseline of results, I just sent normal encrypted email (using Enigmail in SeaMonkey) from my university's mail servers to my ProtonMail inbox. For all the tests I tried with 6 different encodings and appropriate sample texts that included some characters also included in ASCII and some encoding-specific characters. After the email arrived at ProtonMail, I looked at it to see if the text is still readable without any errors.

Encoding	Web mail 4.0.15	iOS app 1.15.11	Bridge to SeaMonkey
ISO-2022-JP	❌	❌	✅
Shift_JIS	✅	❌	✅
EUC-JP	✅	❌	✅
EUC-KR	✅	❌	✅
Big5	✅	❌	✅
GBK	✅	❌	✅

As you can see, when the Bridge is used, all emails were just fine. That's good. It means I'm probably not doing something horribly wrong. The ProtonMail iOS app wasn't able to render any emails correctly, but the ASCII portion was still visible. For the web mail client, the outcome was even more interesting: every encoding except ISO-2022-JP seemed to display just fine. That explains why the ProtonMail support wasn't able to reproduce the problem in the Web client when I sent them a test mail earlier (I used Shift_JIS in my test mail).

Official SimpleLogin service

Next, I continued the tests but now with using the official SimpleLogin instance hosted at simplelogin.io. I didn't change anything about the rest of the setup or the contents of the emails. These are the results:

Encoding	Web mail 4.0.15	iOS app 1.15.11	Bridge to SeaMonkey
ISO-2022-JP	❌	❌	✅
Shift_JIS	❌	❌	✅
EUC-JP	❌	❌	✅
EUC-KR	❌	❌	✅
Big5	❌	❌	✅
GBK	❌	❌	✅

Using the Bridge still works fine. It's safe to say therefore that the emails haven't become completely garbled just by using SimpleLogin. This phenomenon led me to my earlier statements about SimpleLogin having nothing to do really with this issue. However, it seems that SimpleLogin at least has some influence. Maybe it's just because now Enigmail is out of the equation or it has something to do with the additional signing, but somehow none of the emails can be displayed now in the Web mailer. Another curious thing happened: in the iOS client, the emails are now not rendered at all, not even the ASCII portion. It's just completely blank.

Self-hosted SimpleLogin instance

I also have setup my own SimpleLogin instance last week. (Huge thanks to everyone who makes this great open-source software possible!) My server is running Ubuntu 20.04 and I'm using Amazon SES to relay outgoing email, but that should probably not make a difference because the encryption already happens on my server. Results:

Encoding	Web mail 4.0.15	iOS app 1.15.11	Bridge to SeaMonkey
ISO-2022-JP	❌	❌	✅
Shift_JIS	❌	❌	✅
EUC-JP	❌	❌	✅
EUC-KR	✅	❌	✅
Big5	✅	❌	✅
GBK	✅	❌	✅

I didn't really have a good explanation for this at that point, but somehow now the Web mail client was able to render a few of the encodings, but still struggled with some others. None of the emails are fundamentally broken, though, as is evidenced by all of them still being fine when loaded through the Bridge. The iOS app now shows the ASCII portion of the emails again fine. Maybe not signing them makes a difference here, or I'm running slightly different configurations of Postfix etc. that could influence results.

Hexdump analysis

By now I was thoroughly confused, so the next thing I did was download the encrypted emails from the ProtonMail servers so that I could decrypt them locally by myself and then have a look exactly at the binary content. Here are the fundamental differences between the emails:

• Emails sent through my self-hosted SimpleLogin in Shift_JIS and EUC-JP encoding end up as ISO-2022-JP encoding at ProtonMail. Also the other 8bit encodings EUC-KR, Big5 and GBK get a content-transfer-encoding of "base64". I suspect this is something in my server's configuration that tries to ensure compatibility with 7bit SMTP servers. It doesn't really matter for content that gets encrypted and put into ASCII armor later, but it explains most of the differences in testing between the official service and my self-hosted setup
• Emails that are sent directly have a content-transfer-encoding of "quoted-printable". It also happens for ISO-2022-JP, which obviously wouldn't really need it because it's already just 7bit. The culprit might be Enigmail, because any of the mail servers wouldn't be able to change the content of the cleartext email after local encryption. I don't think SeaMonkey is doing it because it doesn't do it when sending plaintext email with the same encoding and email content.
• Emails sent through the official SimpleLogin instance are indeed not touched in any way. The encrypted content is exactly byte-for-byte the way it was sent by my MUA.

And that's it. Even though I understand now that the content of the encrypted emails being slightly differently encoded apparently leads to the ProtonMail web client rendering them in a different way, I still don't understand why this even happens in the first place. Why is it, when I send the exact same emails, that they end up in different encodings with different transfer encodings? If I have some time next week, I'll be able to do a next round of tests where I don't rely any longer on SeaMonkey but speak directly to the SMTP servers via Telnet to test this in a more controlled way and hopefully this leads to some better results. Even though this seems to be fundamentally a problem with ProtonMail, there might be some way to handle the emails at SimpleLogin that could give better compatibility.

[AstrNexus]

Have a similar encoding problem here.

In the case of using PGP, all Chinese text will become garbled. Although there is an error in the text encoding of the content, the text encoding of the sender's name is correct.

I stopped working on the migration to SimpleLogin, so I cannot provide other samples. Also, I have not had similar problems when using ProtonMail to receive PGP encrypted mail from other mail alias services with same original mail sender.

[nguyenkims]

@AstrNexus would you mind trying the same email on other mailbox services to see if the encoding problem is there? This'd help us to narrow down the issue.

[AstrNexus]

I have tried to send the same email to Proton and Outlook with PGP encryption. (Read outlook mail by Thunderbird, create PGP key pair by Thunderbird) Both sides have the same encoding problem.

I also noticed that the headers inserted in the message content by SimpleLogin are encoded properly in ProtonMail iOS client

[nguyenkims]

@AstrNexus thanks for the investigation. Can you send us the email to hi[at]simplelogin.io that causes this issue so we can reproduce and debug?

[AstrNexus]

@AstrNexus thanks for the investigation. Can you send us the email to hi[at]simplelogin.io that causes this issue so we can reproduce and debug?

I received unencrypted mail in Outlook via SimpleLogin and forwarded it to hi[at]simplelogin.io after deleting my personal information.

I have sent to my SimpleLogin alias for testing and confirmed that even if the mail is forwarded from Outlook, the encoding error still occurs on Proton and Thunderbird.

[BadCo-NZ]

ok ill try disabling PGP and see what happens, strange its in mandarin though right? what would cause that?

of course, its www.trademe.co.nz

I have the same issue with SimpleLogin, Protonmail and Trade Me, but only with webmail, it works fine with the Android app.

[salixh5]

I think now that this is mostly an issue with the ProtonMail clients. It's easy to reproduce this by trying to send mails in different encodings directly to your ProtonMail account. Just make sure to at least sign them with PGP (otherwise ProtonMail will convert them to UTF-8 before dropping them into your inbox, and the clients will work fine).

For SimpleLogin an easy fix would be the possibility to disable PGP signing. Also you would need to disable PGP encryption if you have that set up. Then the mails should probably display fine in the ProtonMail clients.