v14 - Mbox Import/Export Testing and Feedback

[cleidigh]

This thread is focused on mbox issues for the new v14 implementation.

@LamKarThess We can continue here...

First goal getting 50 email folder that exports 50, but re-imports with far fewer. Then we will do inspection. @cleidigh

[cleidigh]

@alexeski Thunderbird uses no prefix on mbox files and IETNG has always done the same. I don't want to mess with the ui as I am already way late with this release, however, what I can do is add the preference and you can set it once manually. Then in the next point release I will add it to the ui.

If you can help with some more testing of mbox import and export that would be really helpful. I am looking for big imports and exports as well as odd or old mbox imports and any performance times. Any brute force stuff will help me release soon!

Thanks @cleidigh

[alexeski]

I am looking for big imports and exports as well as odd or old mbox imports and any performance times.

tried to export with a couple of large and old mailboxes, all good!

[cleidigh]

Excellent. Any speed observation especially on import like time to import 1GB or something large? Christopher

On Sat, Sep 2, 2023 at 6:58 AM alexeski @.***> wrote:

I am looking for big imports and exports as well as odd or old mbox imports and any performance times.

tried to export with a couple of large and old mailboxes, all good!

— Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/432#issuecomment-1703801592, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGAZ76565EUJTWVGVMGIC6DXYMGO7ANCNFSM6AAAAAA4GNCJOI . You are receiving this because you were mentioned.Message ID: @.***>

[alexeski]

Any speed observation especially on import like time to import 1GB or something large?

Just tested a 1.5GB mbox file import, took a couple mins or so, reasonable for my underpowered laptop. Worth mentioning I don't have frame of referrence, as I literaly just started using your plugin (with b6, b7, b7a).

Data also looks good to me!

[ArvinZaker]

I imported 7.6 GB of emails from one of my old email accounts. It took about 3 minutes with ~5% CPU activity. Granted, I have a 1TB 980 SAMSUNG EVO SSD and a Ryzen 9 5900X.

There is no data loss, as far as I see. All my files and attachments from the emails are present. The emails are indexed correctly and searchable by Thunderbird.

[cleidigh]

@ArvinZaker Well that is excellent to hear! I assume this was a structured as we do have a 4GB limit per mbox due to the file api. Your speed sounds great , probably a faster than my dev machine. Thanks! @cleidigh

[cleidigh]

@alexeski That's still not bad for an old laptop. BTW, I assumed you were an old time use, so welcome to the ImportExportTools family! Cheers @cleidigh

[alexeski]

I assume this was a structured as we do have a 4GB limit per mbox due to the file api.

ahhh, ok, I was about to report that some large files aren't importing, this explains. I'd like to suggest you throw some feedback to the user about this, so they don't sit down and wait for minutes for nothing (like me :D )

In the same vain, another suggestion both for the import & export, it would be good to show a message in the taskbar that Export completed succefully. (we can see the mbytes processed and it rightfuly disapears after some time, but if we're watching it, perhaps would be interesting to understand when it's really done/successfull?)

[cleidigh]

@alexeski Yes I have not done all the feedback including flagging mboxes which are over 4GB. So you have some larger than that? I have some ideas on how to go about importing larger than 4GB but it won't happen for a while.

Handling the final status msg is also on the todo list. I think it will be a summary and stay for a while. There is just so much going on that I have been focusing on stability. And now from another user I have to figure out why messages from an import don't show up in a a search. @cleidigh @cleidigh

[cleidigh]

Hi! I`m a first time user :)

I used version b7a to import 748 mbox messages into a local folder (they were in maildir, I converted them into mbox using https://github.com/bluebird75/maildir2mbox). It seems to have worked fine! Thanks for the tool and for the beta versions, I am on TB115 and the official releases don`t work there.

However, I realized that the imported messages don`t appear in search results unless I open them first. After quitting thunderbird and launching it again, they do appear, though.

@ericonr

I reproduced your issue with an import not showing up in a search. What I see is different than you, even when I restarted Thunderbird it would not search on the import. So this is an important issue to fix and is one of those things that is tricky with this process. Thank you for finding! @cleidigh

[MidGe48]

Unfortunately, importing mbox does not seem to be working anymore. I tried with the existing imported mbox in thunderbird. Then I tried after deleting the existing mbox in thunderbird, all to no avail.

The issue is that when I select from the top down menu in thunderbird from the Local Folder position: ImportExportTools NG/Import mbox Files/Individual mbox Files it does nothing and does not allow me to select or enter a file path and name.

I also tried the other three menu alternatives (all mbox from a directory, ... ) but all gave me the same: nothing to select and no ability to enter anything. As if the menu pointed to nothing.

Hope this help

[cleidigh]

@MidGe48 hmm,odd...

Clarifications : You are using b7a? Was it working at all at some point? When you say you are on local folder, you mean a folder under the local folders account? Can you check the console log? @cleidigh

[MidGe48]

@cleidigh

yes, b7a and thunderbird 115.1.1

It worked the first and only time I used it immediately after install at the date of my earlier post.

I mean "Local Folders" selected in thunderbird and trying to import the mbox file under that heading.

Console log??? Where do I find that?

[cleidigh]

@MidGe48 Ok first create an import folder under the Local Folders account, then right click on the new subfolder and try to import there. The import and all but export account should not be visible or possible when clicking at the account level, else that's a menu bug. For the console use Control-Shift-J clear window with trash then do operation and check window for errors. @cleidigh

[MidGe48]

@cleidigh

OK, I did as you suggested and created a mbox folder first under Local Folder in thunderbird.

I had the same issue as before the menu shows but does notshow a folder hierarchy to allow to pick a source. It shows nothing.

The journal, with a number of errors is as follows:

services.settings: Failed to load last_modified.json: TypeError: NetworkError when attempting to fetch resource. Utils.sys.mjs:322 This page is in Quirks Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. background.html unreachable code after return statement mboximport.js:1122:1 ioTest worker startup importMboxTest.js:8:9 Trying to load /usr/lib/thunderbird/libotr.so OTRLib.sys.mjs:65:11 Trying to load libotr.so from system's standard library locations OTRLib.sys.mjs:65:11 Trying to load libotr.so.5 from system's standard library locations OTRLib.sys.mjs:65:11 Successfully loaded OTR library libotr.so.5 from system's standard library locations OTRLib.sys.mjs:73:13 TypeError: console is not a function 2 messengerOL.js:419:11 Start backup check sendRemoveListener on closed conduit languagetool-mailextension@languagetool.org.11 2 ConduitsChild.sys.mjs:108 sendRemoveListener on closed conduit languagetool-mailextension@languagetool.org.12 2 ConduitsChild.sys.mjs:108 Missing resource in locale fr: devtools/client/toolbox.ftl Uncaught TypeError: this._buttons is undefined getButton chrome://global/content/elements/dialog.js:190 doEnabling chrome://messenger/content/newFolderDialog.js:80 onLoad chrome://messenger/content/newFolderDialog.js:48 newFolder chrome://messenger/content/about3Pane.js:2915

chrome://messenger/content/about3Pane.js:6113 doCommand chrome://messenger/content/mailCommon.js:677 onCommand chrome://messenger/content/about3Pane.js:590 handleEvent chrome://messenger/content/about3Pane.js:279 dialog.js:190:7 tb.ui.interaction.message_display - The key length must be limited to 72 characters. inserted status ietngUtils.js:110:13 NS_ERROR_NOT_INITIALIZED ietngUtils.js:99 ​Hope this helps, M.

[cleidigh]

@MidGe48 This is very odd, it's as if the menus have been internally munged?? Now the menus are using the new menu api, if you are a prior use you will have noticed I completely redesigned. Now their are two issues that have surfaced with menus. First another add on Cardbook messes with menus causing duplicate entries and other issues. Second I have one report of duplicate menus on Mac with no other addons.

Neither are IETNG issues. So I would disable all addons as one general try. Also doing an uninstall restart and reinstall. Trying the other menu items out would be another test. Sorry this is totally odd never been reported before and sounds like an api issue.

Update : I also did a sanity check under the fr locale just to make sure there wasn't any strange locale scenario, there was not. I did an individual mbox import with no odd menu issues.

@cleidigh

[MidGe48]

@cleidigh

OK, I think I found the issue.

For my use case, when I start my workstation by default it loads a session as it was when last closed.

It seems that is how it looses the menu information.

Closing and restarting thunderbird makes it work as it should. So that seems to be the workaround for now. Not ideal but at least it works.

The issue is thus probably not related to IETNG, but to the linux distro, KDE or wayland, perhaps.

Thanks for IETNG. :)

[TheMartianObserver]

Hello! Found my way here after upgrading to TB 115.2.0 and realizing I needed to update to IETNG beta.

MBOX import is broken for me at the moment using the latest beta b7a with TB 115.2.0 on macOS. On import, TB creates a folder representing the MBOX file but the folder is empty (however, TB properly reports the size of the folder).

Going to my TB profile folder (in Finder), I see the imported MBOX. Looking at it, it looks like the import process munged ALL of the "From" lines, including the ones that normally separate messages. For example, the very first line of the imported MBOX data is >From email@example.com <date/time>.

And I'll echo: Thanks so much for this tool. Couldn't maintain my workflow without it!

[cleidigh]

@TheMartianObserver The mbox import is the one feature blocking release and driving me crazy! Your report goes to the From escaping that's new over prior releases. While I am having issues with the db indexing, this total munging of all the From separators is a first report... Would you be willing to do some debug and sharing of data (privately) to help resolve? I unfortunately can't fix what I cannot see. Thanks @cleidigh

[TheMartianObserver]

Would you be willing to do some debug and sharing of data (privately) to help resolve? I unfortunately can't fix what I cannot see.

@cleidigh Absolutely! I think I can make a sample MBOX file with an appropriately scrubbed email set :) How should I contact you with more information?

[cleidigh]

@TheMartianObserver Excellent!! Use: test1@kokkini.net

Some questions : What is the source of these mbox files? Have they been through any export import process before? Have you used them with a prior IETNG? Thanks in advance. @cleidigh

[TheMartianObserver]

Ug. This is going to take longer than I thought. I originally tried imported an MBOX using exported work emails (which is causing problems), but when trying on test emails from my non-work accounts, I can import MBOX files just fine :-/ I can't send you the work emails as is, but I'll keep playing with it and try to generate an mbox that fails and that I can send along. Unfortunately, all of my tests aren't being done from the same computer, although software versions are identical.

What I've tried: create mbox (using EagleFiler, see below) by merging eml files (originally exported using IETNG 12.0.2), and import that resulting mbox back into TB (115.x) using IETNG v14. I did the same process using emails exported from my work account (Exchange on the back end) and using emails exported from my personal email (IMAP).

To answer your questions:

What is the source of these mbox files? So far, I've tried exports from TB using IETNG (both v12 and v14) and mbox files created by a Mac application called EagleFiler, that can create mbox files by merging eml files. Cursory checks suggest that the mbox it is producing are fine.

Have you used them with a prior IETNG? No, but I can test that out. I just started looking at a different email archival approach, which just happened to coincide with the TB update ¯\_(ツ)_/¯

Have they been through any export import process before? Just to eml for archiving. I was thinking of merging the emls into mobx files for easier handling which is how I got here :)

Thanks! I'll try to get some testing done over the next day or two.

[cleidigh]

@TheMartianObserver Before you go crazy let's establish some baselines and maybe a few test steps. I have done this with a lot of people and it can end up with a lot of time burned and I don't want to waste your time.

Ok before we tackle: (tell me if I get anything wrong)

• Your work/ Exchange mboxes are the issue
• These mboxes were exported from Exchange and have not been altered by any other program
• The imported mbox has an escaped ( > char added in front of From_) on the first line which should be a normal separator, not escaped
• You are on a Mac
• You are able to do some inspection of mbox files

Assuming the above I suggest the following first steps:

• See if you can open and inspect one of your original, unmodified Exchange files
• Copy the first 30 lines to a text file
• Modify just the email addresses for privacy (everything else should just be headers)
• I think you should be comfortable posting here once scrubbed)
• This will give me a lot to start so you don't have to make mboxes yet

Just to explain the escaping, this is used to make lines in body text distinguished from From_ separators. IETNG is somehow escaping where it should not. Unfortunately the mbox format is quirky, old and fraught with these issues. IETNG v14 attempts to help with both Thunderbird and mbox issues where prior versions did not.

Let me know if this is dooable. @cleidigh

[TheMartianObserver]

@cleidigh Thanks! Yeah, sometimes I'll dive right in a go crazy :)

Your first bullet list is essentially correct. I'll work through the second list and get back shortly!

[cleidigh]

Mboxers

I am still not able to get mbox imports to be globally searchable without doing something post import that produces gloda errors proportional to the number of messages. Note, folder filters work fine.

Please try the following if the import is not globally searchable :

• try Thunderbird restart
• if not working, use Force global search toggle on ietng button.

The errors do not appear to cause any bad or visible side effects with my testing

Please report your test results and any observations.

Christopher

[cleidigh]

@alexeski b8 has mbox extension option added for non structured exports. Note comments above on testing imports.cheers @cleidigh

[alexeski]

b8 has mbox extension option added for non structured exports.

thanks for this @cleidigh addition of mbox extension. On a quick test, looks like it's adding the extension with method "Foder export (mbox)" , but not with "Export Remote Folder", is this expected? By the way, excuse me the newby question: what's the use case for "Export Remote Folder"? It suggests sometimes Thunderbird doesn't download the full data available on the Server for offline/local use (maybe the user limits it to X weeks)?

Note comments above on testing imports.cheers

I tried to import a couple of 1.5 mbox file and then immediately was able to search some emails using the (global?) search bar in thunderbid and it seems to just work as expected. The output of the search did show plain text results, not a pre-rendered set, which looked pretty damn bad, but I guess that's a Thunderbird issue, right? Not sure what kind of test or potential bug exactly you're trying to isolate, but to me it's working great!

[cleidigh]

@alexeski Thanks for the quick feedback! Glad you like the mbox extension (thanks for the suggestion) As you see I decided to go all the way and include it in the ui translations and all...

Regarding Remote export, I actually think this will be deprecated as it now is really not functionaly different than my new export method. The old code really was not clear in its differences.

So your imports were immediately searchable, I just have not had that happen. I hope with the subtle changes in b8 this is common for other users.

Yes the search results are totally a Thunderbird thing. They are not meant to be rendered for brevity and ease of viewing.

Like your profile pic. Mind if I ask where you hail from? Being stuck at home I love to know where in the world my users are! Christopher

[Pirx485]

Hi,

I'm trying to migrate from Evolution to Thunderbird on Linux and need to keep my Mail-Archive. Unfortunately I'm having similar issues when I try to import mbox-Files.

I have tried with various Export-sizes between 10 and 150 mails. I can share some of the broken mbox-files as they don't contain sensitive information, let me know where I can send them.

[cleidigh]

@Pirx485 Thanks for posting. I would be happy to help figure out the issue. You can send to: test1@kokkini.net That would be very helpful. There are lots of ways mbox files can trip one up. I just released v14 to the review Q so hopefully we can figure it out. Note I cannot respond with my test account so if I have to it will be my main email. @cleidigh

[cleidigh]

@Pirx485 I wanted to confirm receipt of your email and two mbox files. I inspected found the following :

confirm... mbox has 150 messages [warning]... Mbox has 48 messages

I then tried to import them. Perfect import, 150 and 48 respectively.

This made me realize something...You never stated and I failed to ask what beta you are using. I think (and hope) you are back at b6, I say this because I have a critical mbox import patch that addresses the parsing of headers just like what is found in your mboxes. You can grab b8 from here:

https://github.com/thundernest/import-export-tools-ng/issues/408

Hopefully I am correct and you will see the same results as myself. @cleidigh

[mbravidor]

Hey, I am currently using b8 and tried to import a couple of mbox files (converted from pst). Whereas all sbd folders are created, message import is a mixed bag. Some mbox files are imported completly, some partially, and for some, no messages are imported at all. I could provide some example files.

[cleidigh]

@mbravidor Thank you for reporting in, sorry for the issues. Anything you can send me would be great, it basically is the fastest and sometimes only way to solve these issues.

You can send to my test account here: test1@kokkini.net

I cannot always respond with that account so I will acknowledge here. Thanks and I will try to analyze as quickly as possible since this is critical to get my new mbox code as solid as possible! @cleidigh

[cleidigh]

@mbravidor Just wanted to check if you are still interested in having me look at your mbox files. I didn't get anything in my test account. @cleidigh

[MidGe48]

Mine is working fine, but I have limited usage. I simply delete the existing folder in thunderbird and import my local mbox once a week. It contains only system notifications.

Thanks for the software.

Michel de Greef @.***

"Our human race is affected with a chronic underestimation of the possibility of the future straying from the course initially envisioned" - Nassim Nicholas Taleb

Sent with Tutanota, the secure & ad-free mailbox: https://tutanota.com

Oct 3, 2023, 6:07 AM by @.***:

@mbravidor https://github.com/mbravidor Just wanted to check if you are still interested in having me look at your mbox files. I didn't get anything in my test account.

@cleidigh https://github.com/cleidigh

— Reply to this email directly, > view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/432#issuecomment-1743837905> , or > unsubscribe https://github.com/notifications/unsubscribe-auth/AAHUI6GXDLP6UQZH7VW7UODX5M3K7AVCNFSM6AAAAAA4GNCJOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTHAZTOOJQGU> . You are receiving this because you were mentioned.> Message ID: > <thundernest/import-export-tools-ng/issues/432/1743837905> @> github> .> com>

[vandermark1977]

I have a big .mbox export from Google Workspace which i would love to try out, but .mbox import is greyed out?

Edit: Had to create local folder first, importing as we speak

Edit 2: import of mbox gmail export worked, but there is no folder structure, alle thousands of emails in one local folder and that is of no use. Which setting should i use?

[Pirx485]

@Pirx485 I wanted to confirm receipt of your email and two mbox files. I inspected found the following :

confirm... mbox has 150 messages [warning]... Mbox has 48 messages

I then tried to import them. Perfect import, 150 and 48 respectively.

This made me realize something...You never stated and I failed to ask what beta you are using. I think (and hope) you are back at b6, I say this because I have a critical mbox import patch that addresses the parsing of headers just like what is found in your mboxes. You can grab b8 from here:

408

Hopefully I am correct and you will see the same results as myself. @cleidigh

Unfortunately I was already using b8, I have tried a fresh install, but no change.

[cleidigh]

@Pirx485 I couldn't figure out why, but I just realized it was post b8 for the fix. I will post a pre release here shortly. My apologies for the flub... @cleidigh

[cleidigh]

@vandermark1977 I have used my own Google takeout for testing. I don't recall a structured output. Even if it does, IETNG only works with Thunderbird's sdb structure. I am not sure what your takeout was, however, if you ended up with all mail in one folder, that means that is how Google exported your mail. IETNG has no way of flattening an import. Any more info you can give me on what your takeout looked like? Mine was two mbox files to not exceed 2GB.

Just to be clear, a single mbox file contains one or more messages, but it does not and cannot ever contain structure by definition.

@cleidigh

[cleidigh]

@Pirx485 Give this a whirl. I just confirmed it with your files. import-export-tools-ng-14.0.0-pre1-tb.zip

Restart after installing. @cleidigh

[cleidigh]

@mbravidor I received your mbox files, thanks so much. Your issue is the same as some others where my parser failed with multi line headers. I fixed this post b8. Please try this pre release build. I tested it on your 49 message mbox, all imported fine. Note you will need a restart after installing. https://github.com/thundernest/import-export-tools-ng/files/12795676/import-export-tools-ng-14.0.0-pre1-tb.zip @cleidigh

[MidGe48]

@cleidigh

I tried import-export-tools-ng-14.0.0-pre1-tb.zip without any issues.

Thanks

[cleidigh]

@MidGe48 Thanks for the confirmation! @cleidigh

[Pirx485]

@cleidigh 14.0-pre1 works perfectly, thanks for the support and your efforts :-)

[cleidigh]

@Pirx485 Excellent! That's what I was expecting. The official release happened today so it would be best to download the release from the addon site. Thanks! @cleidigh

[cleidigh]

@Pirx485 Just so you know I'm deleting your mbox files, thanks for sending me those, that was very helpful. @cleidigh

[cleidigh]

v14.0.0 has finally been released!!

I would like to give everyone a big thank you for your help with this major effort. Please download the release from the addon tab or the website.

I already have a set of items I missed or messed up. I have patches that will appear in beta shortly.

export account structure wrong Zip folder messed - use mbox extension option and this works selecting multiple messages for copy to clipboard does not give warning csv export should do html to text conversion Christopher

[mbravidor]

@cleidigh: the updated version works well and solved the issue. However, I encountered another one. Within the sbd-structure, all folders without mails are skipped - even if they contain subfolders witch are not empty (i.e., contain mails). See example image: https://www.dropbox.com/scl/fi/hv0rfpoh83nxbrxpm8ih1/ex.png?rlkey=ckgz2x00gjwlc58v8h2bq5eq0&dl=0

[cleidigh]

@mbravidor I have several test structures that include empty folders with populated subfolders that I just did a successful import test. So now we need to figure out what is different here. Bear with me this has so many things that had to change for v115.

Can you do a screenshot of the top dir you are attempting to import and screenshots down to where you have the issue. I'm looking to get a clearer picture of the exact tree you are importing. Also is this tree an export from a prior 12.x version? @cleidigh

[mbravidor]

@cleidigh: I sent you the requested screenshos via mail. Exports were created with CubeXsoft Outlook Exporter.