search

thunderbird / import-export-tools-ng

Import Export Tools that supports Thunderbird v68-v128
Other
291 stars 31 forks source link

Compaction corrupts imported EMLs in Local Folders #625

Open roberts1000 opened 2 months ago

roberts1000 commented 2 months ago 
Thunderbird: 128.1.1esr (64-bit)
ImportExportTools NG: 14.1.2
No other plugins installed
Windows 10

I recently started exporting emails in .eml format from my provider, then importing them into Local Folders. The process appeared to work fine at first, but after a few days, I started noticing issues with the imported emails:

Symptoms/Issues

  1. Many of the emails show (what looks to be) the source content, instead of, the correctly rendered email. It's particularly a problem with emails that have attachments. (Using Thunderbird's View > Message Body As > Original HTML has no affect. None of the options change how the email is rendered. Neither does the View > Display Attachments Inline toggle.
  2. Some emails are completely blank.
  3. Sometimes the same email will start blank, then show the source, after clicking away and coming back to the email.
  4. Thunderbird would take much longer to open some emails - which is a sign that it's likely going to have one of the above issue. (This seems to have something to do with how much encoded content is present in the source, but that's just a hunch.)

I think I've traced this back to the compaction process that Thunderbird uses. I can't say for sure that ImportExportTools NG is involved, but the corrupted .eml's are only the ones I imported using ImportExportTools NG.

To Reproduce the Error

  1. Setup Thunderbird as a fresh install.
  2. Add an account for an email address you can use to test.
  3. Install ImportExportTools NG.
  4. Find any large image, that's greater than 1MB in size. (It will be used to trigger the compaction process so larger is better).
  5. Go to Thunderbird's Settings > General > Network & Disk Space > Disk Space option. In Compact all folders when it will save over x MB in total, set x to 1MB.
  6. Now, login directly to the email provider you added and send yourself an email, with the image as an attachment.
  7. When the email arrives, export it as a .eml file from their website.
  8. Go back to Thunderbird and import the .eml ile using ImportExportTools NG.
  9. Select the newly imported email and Detach the attached image. (The Delete option also causes the problem.)
  10. Close Thunderbird, then open it back up. The dialog should open, asking if you want to Compact. Select Yes to compact.
  11. Now, navigate to the email in Thunderbird and view it. It should be corrupted. If it's not, close Thunderbird, open it and go back to the email. It's likely corrupted.

Other considerations

  1. All of the emails that I've transferred to local folders, from online folders, using Thunderbird's Move to menu option look correct. They don't get corrupted when I compact, detach/delete attachments. To give this a shot, simply email yourself the same image file again if needed, then use Move to to move the file to a local folder. Detach the attachment, close Thunderbird, open it back up, compact, view the email ...
  2. I tried this process with email addresses from two major providers. Both get corrupted so it doesn't seem to an issue related to original .eml that's generated.
  3. Many of the emails I imported using ImportExportTools NG open correctly, at the moment, but I suspect Thunderbird hasn't tried to alter/compact them yet so they may yet get corrupted in the future.
  4. If I try to use the Repair button on emails that are corrupted, Thunderbird simply deletes them and they're lost forever.
  5. This issue isn't specific to images. I've seen it happen with Excel and PDF attachments as well.
cleidigh commented 2 months ago

@roberts1000 Thank you for the detailed report. I have not looked into the compact issue yet, however, I had a fundamental error in the EML import on Thunderbird v128. Messages were corrupted on import. If you can try b3 beta it may well solve this. If not I will investigate. https://github.com/thunderbird/import-export-tools-ng/blob/v14.1.3/xpi/beta/import-export-tools-ng-14.1.3-b3-tb.xpi @cleidigh

roberts1000 commented 2 months ago

@cleidigh Thanks for the suggestion. I gave 14.1.3-b3 a shot and it does seem to fix the issue! I'm able to import, detach, trigger the compact process and the imported emails continue to render correctly. It appears the error you found/fix was the culprit.

The only issue remaining is, I now have several emails that are corrupted. I tried to export and re-import one email that was clearly having problems, and one that was still rendering correctly (but was imported using an older version ImportExportTools NG, so it likely has issues). They look the same after the re-import, and, when I run Thunderbird's Repair tool on them, Thunderbird still discards them. So it doesn't look like an export and re-import helps get them back on track. I also tried to export, and then import them back into the original email service, but the service thinks both email files have errors in them.

  1. Are there extra characters I could manually remove, or some other change I could make, in the the corrupted files that could get them corrected?
  2. Or, is the corruption repeatable/detectable in a way that ImportExportTools NG could potentially detect the issue and automatically correct them on a re-import? (Or, potentially, a ImportExportTools NG specific Repair option that I could manually invoke.)
cleidigh commented 2 months ago

@roberts1000 Great, I was hoping b3 would address your issue ! I think without actually having your mbox and investigating it I could not tell you what to look for unfortunately. @cleidigh

katsuos commented 6 hours ago

@cleidigh When multiple EML files are imported, The symptom that the raw data of the first imported email is added to the end of the next email to be processed has been improved. Thank you.