Various issues while importing emails via bridge

[alexforencich]

Expected Behavior

Drag and drop emails from other email accounts in email client to import messages into proton email account via bridge should be reliable

Current Behavior

A number of messages seem to trigger various errors, causing the import to fail.

A selection of errors that thunderbird has reported on behalf of the mail sever:

• invalid rfc5322 message: Sender should not be present if equal to From
• failed to import message: failed to import message: imported message is not fully encrypted (code=36027, status=0: known recovered message, possible duplication

Some messages seem to consistently cause timeouts with no further explanation.

The "not fully encrypted" error reported by thunderbird is accompanied by a log message in the bridge log file:

time="Aug  5 02:36:56.747" level=error msg="Failed to import message" error="failed to import message: failed to import message: Imported message is not fully encrypted (Code=36027, Status=0)"

The other error does not seem to result in a log message.

The timeouts also don't appear to be accompanied by any remarkable log messages, and copying messages one at a time seems to reduce the probability of timeouts.

Version Information

Bridge 3.3.2 Thunderbird 102.9.1

Context (Environment)

Attempting to bulk import messages from other email accounts via the bridge

[alexforencich]

Another error: "required field date not found or empty". Seems like something is extremely pedantic about emails being in exactly the right format, not sure if this is in the bridge or on the proton mail backend. Either way, similar to a web browser, these "non-compliant" messages should be accepted without complaint, similar to a web browser rendering HTML that isn't necessarily perfectly written instead of simply giving up when, for example, tags don't get closed in the correct order.

[alexforencich]

Seems like part of this is just https://github.com/ProtonMail/gluon/blob/dev/rfc5322/validation.go . Seems like this validation should simply be skipped - if it's possible to receive a message that violates these rules, then it should be possible to import such a message. Opened https://github.com/ProtonMail/gluon/issues/386 for this.

[LBeernaertProton]

Hey @alexforencich, Thanks for your report.

Regarding your issues:

invalid rfc5322 message: Sender should not be present if equal to From

This has been fixed in Gluon already, the fix should hopefully make its way in 3.5.

failed to import message: failed to import message: imported message is not fully encrypted (code=36027, status=0: known recovered message, possible duplication

Any chance you could provide us with a sample message that triggers this issue?

"required field date not found or empty".

This is a requirement of the RFC 5322 (section 3.6):

The only required header fields are the origination date field and the originator address field(s) 

[alexforencich]

Well, I have emails that are missing the date field. Similar to web browsers, software handling emails has to be able to handle emails that don't necessarily comply to every aspect of the specification. And I suspect there are some other some other header parsing issues as well, I suspect probably related to emails that don't fully comply with the spec. I know I have seen at least one more similar error message so far, but haven't been able to repeat it yet. I'll report them as I come across them and can get a better idea of what the problems are. Again, it's one thing if we're talking about sending new emails, but in this case we're talking about importing emails that were sent by someone else, were received, and stored in a mailbox, and now it's just a question of importing those emails into proton mail. I guess maybe there is a way to edit such offending emails, but I don't think there is an easy way to do that in thunderbird, and either way it is a major issue for bulk importing messages.

[alexforencich]

And yes, I can provide a copy of an email that's triggering the code 36027 issue. How should I get it to you?

[LBeernaertProton]

Well, I have emails that are missing the date field.

We have tracked the date issue internally as GODT-2847. We will make this more permissive.

And yes, I can provide a copy of an email that's triggering the code 36027 issue. How should I get it to you?

You can forward them to my email address.

[alexforencich]

Sure, where do I find that?

[LBeernaertProton]

You can find it in the git commit history.

[alexforencich]

Got it, message sent.

[LBeernaertProton]

I have managed to reproduce the issue. Internally tracked as GODT-2851.

[friwid]

I can reproduce this. Newly made account on Proton, newly installed Bridge. Windows 11, Thunderbird newest version. I wanted to copy all mails from another mail account to Proton.

[itsjustarumour]

Hi All,

I'm still getting this issue when importing emails using Protonmail Bridge v3.4.2 (br-165) on Ubuntu 23.04 and others.

Any updates on a potential fix? Do you need any sample emails?

[LBeernaertProton]

We apologize for delays, we have started looking into this issue.

[yu-tendo]

Hi everyone,

I also have this issue ("imported message is not fully encrypted). Any updates?

Do you need any sample messages? Thank you!

[yu-tendo]

Hi @LBeernaertProton

Please find attached some example emails causing the following error:

failed to import message: failed to import messsage: Imported message is not fully encrypted (Code=36027, Status=0): known recovery message, possible duplication.

example-1.txt: Just an old e-Mail from Amazon example-2.txt: Multi-part message in MIME format example-3.txt: with an invitation example-4.txt: auto-reply e-Mail

Version Information

Proton Mail Bridge v3.8.1 (br-187) Thunderbird 115.6.0 (64-bit) Microsoft Windows 10

Attachments

example-1.txt example-2.txt example-3.txt example-4.txt

[LBeernaertProton]

@yu-tendo the error failed to import message: failed to import messsage: Imported message is not fully encrypted should be fixed with bridge 3.8.

Did you import those messages with 3.8?

[yu-tendo]

@LBeernaertProton Yes, I used Proton Mail Bridge v3.8.1 (br-187). I can confirm that the number of e-Mails causing the issue ("Imported message is not fully encrypted") got significantly reduced (compared to 3.6.1); however, the e-Mails from above still cause this issue.

The e-Mails from above are exported with Thunderbird. Please just change the file extension from ".txt" to ".eml" (GitHub blocks ".eml"). If you open them with Thunderbird, I hope you should be able to reproduce this issue.

Please let me know if you need additional information. Thank you!

[oussjarrousse]

I am also getting the same error when I am trying to transfer emails using Thunderbird (and the bridge) from an email box outside of protonmail to a mailbox on protonmail. It happens for some messages...

Here is the error (notification)

"failed to import message: failed to import message: imported message is not fully encrypted (code=36027, status=0: known recovered message, possible duplication"

FYI, when I use Mail on Mac OS X to do the same thing, I don't have this issue, those messages get transferred without problems.

[HanaaDany1]

I am having the issue as well (TB 115.11.1 and Bridge v3.7.1 as well as TB 78.14.0), any workaround for this issue? Thank you so much for following up on this!