Critical issue: EML import functionality completely broken

[de04080]

Description

Importing messages in EML format is broken. I have exported messages from Outlook.com, iCloud.com, and other services. I initially reported this issue in bug report #546, but after reviewing the comments, I realized that the problem is related to endless header content being added to the messages, during the import of EML messages back into Thunderbird. Therefore, I am opening a separate bug report for this issue. Thank you.

System Information

• Thunderbird 128
• Import Export Tools NG 14.12
• Windows 11 Pro

Exporting to

• MBOX = fine
• HTML = fine
• Plain Text = fine
• EML = broken!

Details

A folder containing 10 emails and 2 attachments is 286 KB. After exporting it to EML and re-importing it, the size increases to 3.9 MB. Most of the messages are cluttered with extensive header content, as shown in the attached screenshot below, and they no longer resemble the originals.

None of the options in the add-on made a difference. Exporting each email individually versus exporting the full folder did not change the outcome. Threading and unthreading the emails before exporting also had no effect.

[de04080]

I have edited the description and believe that the import feature is broken. I exported emails to both MBOX and EML formats. When I viewed the content in VS Code, it looked similar. However, when importing them back into Thunderbird, the MBOX format strips out the headers and only displays the message bodies, whereas the EML format displays everything, including headers, which it should not.

[de04080]

Here's another example of what appears after the email closing when I import an EML message in Thunderbird.

[cleidigh]

@de04080 I have a beta for this issue. So far my testing looks good doing imports up to 20k eml messages. Let me know how it looks on your side.

https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b2-tb.xpi

@cleidigh

[de04080]

@cleidigh Thank you for the reply. I saved the .xpi on my system, but I have an error message when I use Install Add-on from File in TB.

The flag xpinstall.signatures.required is set to false.

[cleidigh]

@de04080 This is a github quirk. See below for tip https://github.com/thunderbird/import-export-tools-ng/issues/595 Also if you download with Firefox it will try to install it. Make sure you go to Thunderbird addon tab, gear icon, install addon from file.

@cleidigh

[de04080]

@cleidigh I have installed ImportExportTools NG 14.1.3-b2-tb.xpi.

I exported an email folder to EML and don’t see any abnormalities when I import the folder back into Thunderbird.

However, the size of the email folder before export was 270 KB, and it increased to 703 KB after import. Ideally, I would expect the size to be the same before export and after import. Is this normal?

I exported the same folder to both EML and MBOX formats. When I import them back, they both have the same size of 705 KB.

[kmbuthia]

Hello @cleidigh

I also installed this "patched" version of the addon and can confirm that bulk import eml files no longer works. It did some weeks ago but not any more unfortunately. At the moment, uploading one eml file at a time is the only thing that seems to work. If there's another bulk import option/approach that can be used as a work-around, please advise and thanks a lot in advance for the guidance/support.

[cleidigh]

@de04080 You are correct, you should not see a more than double difference. The mbox after import will be slightly larger with some additional headers, but minimal. So far I cannot reproduce what you see. I did multiple export imports with small and multi megabyte folders all with expected results. Can you try just a few messages and see if you get such different sizes? Then maybe we can analyze. @cleidigh

[cleidigh]

@kmbuthia Can you id what Thunderbird version? v14.1.3-b2 should fix for Thunderbird v128, however, it won't work on v115. I am doing a rev that will work on both now. Can you be more specific about what is happening? What was working before? @cleidigh

[kmbuthia]

@cleidigh Thunderbird version: 128.1.0esr (64 bit - on a Mac)

Steps to reproduce eml import failure:

1. I exported a mailbox with this plugin and chose eml as the export option (roughly 100 email threads. Total size less than 1GB)
2. I then attempted to bulk import the eml messages into a new inbox using the eml import option

Issue/error: I was unable to import more than one eml message at a time. Any attempt to use the "import all eml messages from a directory" or "import eml message from a directory with subdirectories" ended up importing one message then hanging indefinitely after that (waited 30min before killing the operation). As mentioned before, what did work was choosing the "import individual eml messages" option then choosing a single eml file. Even with this option choosing just two emls would result in one eml being successfully uploaded and then the process hangs and the second eml doesn't get imported.

Hope this helps and let me know if I can provide more info.

[cleidigh]

@kmbuthia So I'm still trying to understand if importing multiple eml messages ever worked with v128esr? I see that you did install v14.1.3-b2 and that was not working. It might be that you were trying to import to an imap folder? Try a local folder With b2.

Can you also try beta 1? https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b1-tb.xpi

I may have to add debug as this should work. @cleidigh

[de04080]

Hi @cleidigh,

I have a Thunderbird folder that is 270 KB in size with 9 messages.

1. I exported All Messages In Folder > EML Message Format.
2. I exported EML Message Format (each message individually).

I created two local folders, "Test 1" and "Test 2":

1. I imported all EML Messages From a Directory into Test 1 (point 1 above).
2. I imported Individual EML Messages into Test 2 (point 2 above).

Both methods of export/import worked, but there is a noticeable difference in the file sizes for each method. Please see the attached screenshot for details.

[de04080]

@cleidigh I conducted another test with a different folder that is 450 KB in size. I exported it in both EML and MBOX formats. When importing both formats, the resulting file sizes were identical, at 1.1 MB. It's worth noting that these emails are not simple text but rather complex Microsoft Outlook HTML (I'm not entirely sure of the exact terminology), which consists of a substantial amount of HTML.

[cleidigh]

@de04080 This still perplexes me. I really think I need an example of your inputs and import results. Can you try to identify a couple of non personal eml messages that when imported are larger. Then if you can send me those and the resulting mbox to my test account test1@kokkini.net

To get the mbox you can choose Open folder directory and locate the mbox for the folder where you imported the couple emls. The mbox will have no extension. This way I can really compare things. @cleidigh

[aryanrajseo]

Hi, i am trying to import eml files and it import 1 and then got stuck. Using Win 11 Pro 22H2, latest version.

[sinnfeinn]

Hi, i am trying to import eml files and it import 1 and then got stuck. Using Win 11 Pro 22H2, latest version.

Same issue

[kmbuthia]

The screenshots above is also what I experienced @cleidigh while trying to use the import.

I just noticed that @de04080 screenshot shows uploading to local folders. However in my case (and I suspect it's the same case for sinnfein and aryanrajseo) we are trying to upload/import to an email server, not a local folder...and that doesn't seem to be working.

Let me know if this helps or if the plugin only supports import to local folders.

[cleidigh]

@kmbuthia Did you try b2, that addresses both local folders and imap. https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b2-tb.xpi @cleidigh

[sinnfeinn]

@kmbuthia

Did you try b2, that addresses both local folders and imap.

https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b2-tb.xpi

@cleidigh

Yes! b2 I am able to export mbox for outgoing but not for inbox. Also every eml export with folders ends immediately. Import stops after 1 eml.

[cleidigh]

@kmbuthia Working on it... @cleidigh

[kmbuthia]

It might be that you were trying to import to an imap folder? Try a local folder With b2.

Thanks @cleidigh and yes to answer the above question you asked some time ago...I was indeed trying to import into an imap folder/email server. Local folder import certainly works...but upload to IMAP server does not work unfortunately.

[cleidigh]

@kmbuthia @sinnfeinn @de04080 Ok I just posted b3 with a specific fix for imap eml imports. I don't know why I missed this, but I did. Besides various individual imports, I did a folder import of over 1500 messages. Since it's imap it's slower depending on your server etc.

https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b3-tb.xpi

@cleidigh

[de04080]

@cleidigh I have tried version 3, and as mentioned by @kmbuthia, I am using a local folder. I did not notice this bug before.

Regarding the email size difference, it's still the same.

Screenshot 1 shows Outlook HTML messages in Thunderbird (In Plain text).

Screenshot 2 shows Outlook HTML exported as EML (full folder) in Widows 11.

Screenshot 3 shows Outlook HTML exported as MBOX files In Windows 11.

I do not have non-personal messages, but I could send those if you assure me that you will not use the content in any manner and will destroy them. However, how should I send them? If I send you either the EML or the MBOX files, their size will be inflated.

[kmbuthia]

https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b3-tb.xpi

@cleidigh some good news on my end.

The last patch you shared above does indeed work. I am able to import multiple emails once more into an email server (IMAP server). Ran a test with ~250 emails and they all got uploaded smoothly.

Thanks a lot for the fix! 👍🏽