Import All Messages and Sub Folders

[danielehrhardt]

Hi,

i think the Import Feature for EML Files with Sub Folders is not working. I starts to Create one Folder and then it stops.

Best, Daniel

[cleidigh]

@danielehrhardt there does seem to be an issue with some large imports. I have not been able to fix it given its inconsistency in complexity. What OS/versions are you using?

[RulerOf]

I'm experiencing the same problem using Thunderbird 68.6.0 (64-bit) on Mac OS 10.14.6. The Addon version is 4.0.4.

[cleidigh]

@RulerOf Can you identify more details? Do you see any errors in the console output? are you seeing any specific behavior mentioned above? There are potentially a couple of issues here I have been working on for some time without resolution. Any additional information is usually helpful. Sorry for the issues... @cleidigh

[RulerOf]

@cleidigh I should have thought to look for the error console :)

NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.getChildNamed] mboximport.js:1048
    buildEMLarray chrome://mboximport/content/mboximport.js:1048
    RUNimportALLasEML chrome://mboximport/content/mboximport.js:1017
    importALLasEML chrome://mboximport/content/mboximport.js:1010

These files are coming from a mounted Samba share. I'll bring them local and try again when I get the chance later today, but I don't expect it to make a difference.

Edit: No change importing from local storage.

[cleidigh]

@RulerOf Thanks. that is a slightly different error then I have seen before. Do any folders get created, anything imported? The samba share may indeed be an issue. I have seen several comments on some issues with Thunderbird and just Windows 10 (I know you are under OS X, still maybe an issue) @cleidigh

[RulerOf]

@cleidigh One folder gets created, no messages imported. I also got a different error once but it seems to have been a fluke. I can reliably reproduce the 0x80004005 error from above.

Here is a video of my latest import attempt.

Also, is this known not to be a problem on a specific OS? I have a Windows 8.1 VM I could run this on instead. Should I give it a try there?

Edit: Using Windows 8.1 didn't make a difference, and I get the exact same error.

[RulerOf]

@cleidigh can you suggest an alternative method to import an EML folder structure into an IMAP account, and/or can you provide an estimate for sponsoring a fix to this problem?

[cleidigh]

@RulerOf sorry for the delay been dealing with a lot of things lately. is there any possibility of doing a zoom call? I have to do everything by dictation as I am disabled this is a quirky issue I believe related to several others I have not been able to fix. if I can get the minimal skeleton structure of your import (without the actual e-mail) I may be able to reproduce and have better luck debugging where you located? as soon as I to the next Release (hopefully this week) I will be adding a donate button to the homepage. then I will be focusing on experiments to solve this and the related issues.

I don't think I asked for . if you try importing a single folder to see if that will work @cleidigh

[obar]

Hi @cleidigh!

I've seen you mention the concept of importing with structure in other issues. This seems to be where I'm running into a problem, importing mail from an Apple Mail export (and I suspect the topic of this issue). Right now the best I can do for importing a nested structure is to right-click on the target location for the import, select Import mbox file, and in the resulting dialog box choosing Select a directory where searching the mbox files to import (also in subdirectories)

The result I get is a flat structure with many folders called mbox[number] (with one entry not having a number, and other numbers not sequential, 1 to 3 digits in length, and not consistent across imports of the same folders to different target folders). The folder names and structure are lost. If I try Import all messages from a directory>Also from its subdirectories, the structure is there and the names are kept (albeit they exist with the .mbox extension in the folder name, as you'll see below the reason for that makes sense), but the messages are not there (understandably, because this is looking for email files and not mbox).

Here's the structure of the export from Apple Mail on OS X High Sierra for an exported folder:

ExportedFolder
├── ExportedFolder
│   ├── Nested1.mbox
│   │   ├── mbox
│   │   └── table_of_contents
│   ├── Nested2.mbox
│   │   ├── mbox
│   │   └── table_of_contents
│   └── Nested3
│       ├── Child.mbox
│       │   ├── mbox
│       │   └── table_of_contents
│       └── Nested3.mbox
│           ├── mbox
│           └── table_of_contents
└── ExportedFolder.mbox
    ├── mbox
    └── table_of_contents

Should I be using a different import entry point than what I described above? Perhaps there are settings I haven't seen?

Thanks for your help!

[cleidigh]

@obar The import with subdirectories (structure) has limitations in terms of what it expects. I have some notes on each of the options in the following post: #41 it gives some clarity which will also end up in my user guide . eventually. The problem with AppleMail is that the directory structure creates an export uses the container folders as names but the mbox files under them are just named mbox. Currently the extension . test not recognize the parent folder name. The only real structured import is with structures using .sbd subfolder names. I have this on the to do list, however, I am not sure when that will be.

The (not very great) workaround is either: 1 - create the folder structure within Thunderbird first (empty), then import the individual Mbox files to each folder. 2 - change the folder structure to use .sbd subfolders

I will open an enhancement issue for this let me know your progress @cleidigh I am just about to have 4.1.0 released but this is too big to squeeze it in now.

[obar]

Thanks @cleidigh!

This is a pretty sizable import in terms of folder count that I'm helping out with (not my mail), so workaround 1 isn't viable. I am therefore curious about 2, which perhaps could be transformed in a renaming script.

Am I correct in understanding that the input and output structures of such a script would be something like this?

Apple export style:

ExportedFolder
├── ExportedFolder
│   ├── Nested1.mbox
│   │   ├── mbox
│   │   └── table_of_contents
│   ├── Nested2.mbox
│   │   ├── mbox
│   │   └── table_of_contents
│   ├── Nested3
│   │   └── Child.mbox
│   │       ├── mbox
│   │       └── table_of_contents
│   └── Nested3.mbox
│       ├── mbox
│       └── table_of_contents
└── ExportedFolder.mbox
    ├── mbox
    └── table_of_contents

sbd style:

ExportedFolder
├── ExportedFolder.mbox
└── ExportedFolder.sbd
    ├── Nested1.mbox
    ├── Nested2.mbox
    ├── Nested3.mbox
    └── Nested3.sbd
        └── Child.mbox

[cleidigh]

@obar since your post I have been thinking the same thing about a script. I should probably put together either a Python or JavaScript. script to do this until I can support it directly. I have so many things in my hand I have forgotten some of these issues. I think you drop the extension on the actual mbox files.

[obar]

I just tried it out on a small scale, and you're absolutely right: the sbd style does not take file extensions on the mbox files and should be:

ExportedFolder
├── ExportedFolder
└── ExportedFolder.sbd
    ├── Nested1
    ├── Nested2
    ├── Nested3
    └── Nested3.sbd
        └── Child

I'm happy to make the script and contribute it in the next couple of days. I very much appreciate your work on the plugin and your help with my questions.

[cleidigh]

Great! I am on the fence regarding scripting approach. what were you thinking? I would lean towards JavaScript however Node is not installed by default, but it is the web scripting language of choice.

[obar]

I wrote a simple python script.

Apple ships an old version of python with OS X but even that is coming to an end, so I didn't bother targeting it. This will still need a download for users, but there are small downloads for python across platforms and this script only uses the system libraries.

I'll send it over after some more testing :)

[hfreemanPE]

I have Windows 7 professional TB 68.8.0 32 bit. I used "input all messages from a directory/ also from it subdirectories" the input consisted of a folder with 81000+ .eml and 8 sub folders totaling just under 400,000 .eml. It ran and imported all the folders (empty) and 22180 of the .eml in the main folder and stopped after around 6 or so hours. it seems to be loading a message in little over a second. I got aprox 2500 in an hour so projected to run about 3 1/2 days. have 16 core i9 64G ram ssd etc, so reasonable power. Is this time what i should expect? Should i just restart it? does it duplicate files already imported? or should i break it down into smaller chunks? I thought about importing the sub floders all first then the main folder last with out sub directories. thanks

[cleidigh]

@hfreemanPE cc: @RulerOf @danielehrhardt

are you possibly online? I've been working on this for a while time as there are several issues. you should certainly not be seeing 1 message/s I have recently seen this myself but not consistently. I have been working on completely rewritten sections of the extension to address several issues. This pointer below is to a test extension that does a basic EML import including structure. It installs on the folder context menu at the bottom. Ignore the first two test functions, just use ImportEMLStructuredExt I would leave all the options alone for now.

On my machine test results as follows: (i7 8700, 64GB, very fast SSD)

4 top folders/10 subfolders each/400 messages each : 16000 total messages , 1GB total size Import time: ~130s

As this is a very important problem to solve, any testing help very appreciated! I should be online into later this evening. Thanks Christopher

https://github.com/cleidigh/iet-tests/blob/master/xpi/iet-ng-tests-0.1.1-tb.xpi

[RulerOf]

@cleidigh I'll take a stab at this tonight and report back.

[cleidigh]

@RulerOf That would be great ! I really want to get some progress on this.

• I suggest importing something below 1GB to start
• there is some console output
• probably import to a top-level local folder

@cleidigh

[cleidigh]

@RulerOf I just realized the test may have an issue on OS X because of file paths, I forgot you are on OS X. I am going to try and check and fix if possible now.

[RulerOf]

@cleidigh I'm on OS X but I can try on Win8 just as easily. I'll do both if I have a problem.

[cleidigh]

@RulerOf I have to normalize all of the path work, looking at it now

[cleidigh]

@RulerOf forget OS X for the moment. it appears something with script scopes changed on OS X renders everything inoperable. matches what another user reported. unfortunately I will have to investigate that the Windows version should be okay

[hfreemanPE]

I ran two runs. It imported all 8 sub folders and began filling them. the first folder imported all 98 EML and the second folder imported 9311 the first run and 12974 the second one. Both times i started with a clean folder to import to. TB then crashed. Is there information about the crash i can gather for you?

On 5/21/2020 5:00 PM, Christopher Leidigh wrote:

@hfreemanPE https://github.com/hfreemanPE cc: @RulerOf https://github.com/RulerOf @danielehrhardt https://github.com/danielehrhardt

are you possibly online? I've been working on this for a while time as there are several issues. you should certainly not be seeing 1 message/s I have recently seen this myself but not consistently. I have been working on completely rewritten sections of the extension to address several issues. This pointer below is to a test extension that does a basic EML import including structure. It installs on the folder context menu at the bottom. Ignore the first two test functions, just use ImportEMLStructuredExt I would leave all the options alone for now.

On my machine test results as follows: (i7 8700, 64GB, very fast SSD)

4 top folders/10 subfolders each/400 messages each : 16000 total messages , 1GB total size Import time: ~130s

As this is a very important problem to solve, any testing help very appreciated! I should be online into later this evening. Thanks Christopher

https://github.com/cleidigh/iet-tests/blob/master/xpi/iet-ng-tests-0.1.1-tb.xpi

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632368216, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJOQ2DL2YPREZOG5NU7DRSWP6ZANCNFSM4K3OUFKA.

[cleidigh]

@hfreemanPE you can check to debug output (ShiftControl-J) unfortunately I was hoping for better results. once there are errors those folders can be messed up.

[cleidigh]

@hfreemanPE by the way, how was the speed ?

[cleidigh]

@hfreemanPE you can also try changing the updateCycle to a lower number like 100 in the options. I am hoping the debugging info gives us some clues.

[cleidigh]

@RulerOf I think the OS X issue might be JavaScript caching after the update. I would try starting Thunderbird with -purgecaches command line option that may do the trick

[hfreemanPE]

setting to 100 it ran for about 15 minutes and imported 41843 EML with the larger number of folders with fewer per folder scenario. much better.

however on the eight sub folders with 300000+ it crashes in about 5 minutes with only 3526 EML imported in the second folder. first one is finished correctly as before.

On 5/21/2020 11:50 PM, Christopher Leidigh wrote:

@RulerOf https://github.com/RulerOf I think the OS X issue might be JavaScript caching after the update. I would try starting Thunderbird with -purgecaches command line option that may do the trick

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632478435, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJORBF6H732UJRMKTH4TRSYAAVANCNFSM4K3OUFKA.

[cleidigh]

@hfreemanPE just saw your post before I crash for the night. can you capture and post the debug output? I am hoping we catch something.

[hfreemanPE]

note id did crash in the first run also, just much more imported before the crash

On 5/22/2020 12:05 AM, Harrell Freeman wrote:

setting to 100 it ran for about 15 minutes and imported 41843 EML with the larger number of folders with fewer per folder scenario. much better.

however on the eight sub folders with 300000+ it crashes in about 5 minutes with only 3526 EML imported in the second folder. first one is finished correctly as before.

On 5/21/2020 11:50 PM, Christopher Leidigh wrote:

@RulerOf https://github.com/RulerOf I think the OS X issue might be JavaScript caching after the update. I would try starting Thunderbird with -purgecaches command line option that may do the trick

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632478435, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJORBF6H732UJRMKTH4TRSYAAVANCNFSM4K3OUFKA.

[hfreemanPE]

I am doing this after the restart is that correct, never done this before

XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. messenger.xul WebExtensions: Loading packed extension from F:\for thunderbird profile\extensions\iet-ng-tests@cleidigh.kokkini.net.xpi ext-legacy.js:157 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:3 WebExtensions: Loading packed extension from F:\for thunderbird profile\extensions{e2fda1a4-762b-4020-b5ad-a41df1933103}.xpi ext-legacy.js:157 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:13 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:14 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:15 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:16 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:17 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:18 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:19 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:20 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:21 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:22 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:23 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:24 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:25 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:26 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:27 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:28 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:29 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:30 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:31 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:32 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:33 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:38 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:39 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:40 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:41 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:42 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:43 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:44 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:45 Ignoring unrecognized chrome manifest directive 'style'. chrome.manifest:46 WebExtensions: Loading packed extension from F:\for thunderbird profile\extensions\ImportExportToolsNG@cleidigh.kokkini.net.xpi ext-legacy.js:157 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:7 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:8 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:10 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:11 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:12 Ignoring unrecognized chrome manifest directive 'overlay'. chrome.manifest:13 WebExtensions: Loading add-on preferences from  F:\for thunderbird profile\extensions\iet-ng-tests@cleidigh.kokkini.net.xpi ext-legacy.js:190 WebExtensions: Firing profile-after-change listeners for iet-ng-tests@cleidigh.kokkini.net ext-legacy.js:194 WebExtensions: Loading add-on preferences from  F:\for thunderbird profile\extensions\ImportExportToolsNG@cleidigh.kokkini.net.xpi ext-legacy.js:190 WebExtensions: Firing profile-after-change listeners for ImportExportToolsNG@cleidigh.kokkini.net ext-legacy.js:194 WebExtensions: Loading add-on preferences from  F:\for thunderbird profile\extensions{e2fda1a4-762b-4020-b5ad-a41df1933103}.xpi ext-legacy.js:190 WebExtensions: Firing profile-after-change listeners for {e2fda1a4-762b-4020-b5ad-a41df1933103} ext-legacy.js:194 Use of Mutation Events is deprecated. Use MutationObserver instead. calendar-views.js:97:9 LoadPreferences utils.js:7:10 Object { _all: {…}, _add: _add(), add: add(), addAll: addAll(), get: get(), getAll: getAll(), defaultBranch: XPCWrappedNative_NoHelper, type: Getter, instantApply: Getter, _instantApplyForceEnabled: false, … } utils.js:29:10 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 5 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 messenger.xul UpdateUtils.getAppUpdateAutoEnabled - Unable to read app update configuration file. Exception: Win error 2 during operation open on file C:\ProgramData\Mozilla\updates\D78BF5DD33499EC2\update-config.json (The system cannot find the file specified.

) XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. messenger.xul UpdateUtils.getAppUpdateAutoEnabled - Unable to read app update configuration file. Exception: Win error 2 during operation open on file C:\ProgramData\Mozilla\updates\D78BF5DD33499EC2\update-config.json (The system cannot find the file specified.

) 2 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 16 messenger.xul UpdateUtils.getAppUpdateAutoEnabled - Unable to read app update configuration file. Exception: Win error 2 during operation open on file C:\ProgramData\Mozilla\updates\D78BF5DD33499EC2\update-config.json (The system cannot find the file specified.

) XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 5 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 3 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 5 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 folderDisplay.js:2673:9 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 4 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. mailWidgets.js:991:29 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 calendar-multiday-base-view.js:688:43 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. mailWidgets.js:991:29 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 13 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. mailWidgets.js:991:29 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 5 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. mailWidgets.js:991:29 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. messenger.xul TypeError: aDocumentNode is undefined 5 msgHdrView.js:1535:1 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 13 messenger.xul XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. mailWidgets.js:991:29 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. 2 calendar-multiday-base-view.js:688:43 XUL box for statusbarpanel element contained an inline #text child, forcing all its children to be wrapped in a block. mailWidgets.js:991:29 Stylesheets without CSSOM changes reparsed to check for errors. Refresh the page to also see errors from stylesheets changed from CSSOM and from style attributes. 16 Unknown property ‘user-select’.  Declaration dropped. reps.css:23:15 Stylesheets without CSSOM changes reparsed to check for errors. Refresh the page to also see errors from stylesheets changed from CSSOM and from style attributes.

On 5/22/2020 12:09 AM, Christopher Leidigh wrote:

@hfreemanPE https://github.com/hfreemanPE just saw your post before I crash for the night. can you capture and post the debug output? I am hoping we catch something.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632484112, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJORVDFQVNZEU3QDLUETRSYCG7ANCNFSM4K3OUFKA.

[cleidigh]

ideally after a problem however I think you are reporting that it always crashes at some point so we cannot The debug I will think about some added output crashing now...

[hfreemanPE]

If I do a small amount say 3000 in one folder with 7 subfolders then it will finish correctly but will not import the eml in the top folder, just import the 7 sub folders and they look good. otherwise with the other 2 scenario it ALWAYS crashes somewhere in the process. with the updateCycle set to 400 then the many eml few folders runs better and with100 set then the many folders scenario runs better.

On 5/22/2020 12:15 AM, Christopher Leidigh wrote:

ideally after a problem however I think you are reporting that it always crashes at some point so we cannot The debug I will think about some added output crashing now...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632486127, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJOTPOIVAIEEJ74YN5DDRSYC7LANCNFSM4K3OUFKA.

[cleidigh]

@hfreemanPE If you can describe your exact structure , the ruff amount of messages in each folder as well stay total data size folder. I want to try and replicate set up as close as possible. I think we're bumping up against him dynamic limits working against the database. Also a single resulting mbox file with ~300000 messages may also be an issue, not necessarily that it's not possible rather it may have to be done with different periodic wait times or other methods. The new experimental methods in the extension are not yet optimized/fully debugged. we have to get it to the point where we can catch some form of an error.

[hfreemanPE]

see attached capture for the structure showing the number of messages in each folder. this is taken from the TB as imported but matches the files i am importing on a crash disk from another computer in windows live mail. the number of messages exactly match the source up until the crash. in the case of capture1 it is the many files fewer EML each. it ran down to the point of Casey Research when it crashed before finishing that folder. the ones shown still with no EML are the next level down. I truncated the capture at the failed folder Casey Research. below that all are empty at that level. It was filling them down the page. there is 73000+ in all the folders and it crashed after loading 19726

Capture2 is of the structure with around 50000 each in three of 7 folders. the AAICU has 98 which matches the source and the inbox mar 1 only loaded 3429 of 52758 before crashing.

see the size of each source attached

On 5/22/2020 12:28 PM, Christopher Leidigh wrote:

@hfreemanPE https://github.com/hfreemanPE If you can describe your exact structure , the ruff amount of messages in each folder as well stay total data size folder. I want to try and replicate set up as close as possible. I think we're bumping up against him dynamic limits working against the database. Also a single resulting mbox file with ~300000 messages may also be an issue, not necessarily that it's not possible rather it may have to be done with different periodic wait times or other methods. The new experimental methods in the extension are not yet optimized/fully debugged. we have to get it to the point where we can catch some form of an error.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632819332, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJOR5FIEXM4MMTYC3XJTRS2Y43ANCNFSM4K3OUFKA.

[cleidigh]

@hfreemanPE I don't see any attachment. just give me some of the sizes (total data size's) of some of the biggest folders I will try and generate something similar to test Since Thunderbird is crashing I believe we are reaching resource limits hopefully further inspection will allow me to do something within the extension I am sure nothing happens Thunderbird core anytime soon thanks for continuing to help me on this!

[hfreemanPE]

did you get them this time I sent twice but it is not showing up

On 5/22/2020 6:25 PM, Harrell Freeman wrote:

On 5/22/2020 6:22 PM, Harrell Freeman wrote:

On 5/22/2020 6:19 PM, Christopher Leidigh wrote:

@hfreemanPE https://github.com/hfreemanPE I don't see any attachment. just give me some of the sizes (total data size's) of some of the biggest folders I will try and generate something similar to test Since Thunderbird is crashing I believe we are reaching resource limits hopefully further inspection will allow me to do something within the extension I am sure nothing happens Thunderbird core anytime soon thanks for continuing to help me on this!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632941003, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJOWJPGJI64IRRUCGLG3RS4B6RANCNFSM4K3OUFKA.

[hfreemanPE]

I am trying from another mail account this time

On 5/22/2020 6:25 PM, Harrell Freeman wrote:

On 5/22/2020 6:22 PM, Harrell Freeman wrote:

On 5/22/2020 6:19 PM, Christopher Leidigh wrote:

@hfreemanPE https://github.com/hfreemanPE I don't see any attachment. just give me some of the sizes (total data size's) of some of the biggest folders I will try and generate something similar to test Since Thunderbird is crashing I believe we are reaching resource limits hopefully further inspection will allow me to do something within the extension I am sure nothing happens Thunderbird core anytime soon thanks for continuing to help me on this!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-632941003, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJOWJPGJI64IRRUCGLG3RS4B6RANCNFSM4K3OUFKA.

[hfreemanPE]

did this work?

[cleidigh]

@hfreemanPE got it! did you say you had a single folder with 500,000 individual messages? I believe the maximum test import . I have done is 7GB I have to reproduce the problem here as I need to get access to the debug info. I was not expecting to be optimizing the experiment yet, but we will give it a try.

[cleidigh]

Another question are you using a separate test profile? the reason I ask is that once you get errors in the database you sometimes need to start with a clean profile. Or, I also like to do that for any large-scale tests.

[cleidigh]

Try 0.1.2 it has a small change but might do something I just did an import 50k messages one four

500s

[hfreemanPE]

I am using the same profile

[hfreemanPE]

where do i find 0.1.2

[cleidigh]

@hfreemanPE I am sorry I just realized I never uploaded 0.1.2 and I overrode it locally. I am currently trying to fix the import of the top folder (silly mistake on my part), not sure if I can get it out tonight.

I also created a message generator and did the following with 0.1.1:

• 1 subfolder
• 20k random size messages
• Total data import size : 9.47GB

Import Time: 342s only 36s/GB !!

that is pretty darn fast/no errors

I would like to see if I didn't get you to the same place. Can you run Node JavaScript scripts? the script I have a standalone, requiring Node.JS

In the meantime I am doing more experiments , let's see what happens on your system . if you import just largest of one of your folders.

I want to see if we can determine at what point things slow down before crashing. I never saw any slow down on my machine and the database already have lots of messages from prior tests. I had a similar problem with the another user who was also doing a large import. The first two problems were folder and message paths that were too long. Because these words tell with in the core the exact errors never bubbled up. I am not certain, but we might be dealing with a similar issue.

[hfreemanPE]

how to run Node JavaScript? willing to try but no experience. Doesnt seem to slow down before crash except a pause for second or two every so often, then starts full speed. note on multiple runs it stops at different places.

[cleidigh]

@hfreemanPE I was just about to sign off (I am on the East Coast of the US), I assume you are somewhere ~12 hours difference? I was hoping to be able to post something tonight, but the other things took to long. I was amazed at the speed of the report . test I did today. it does make me more confident that we just have to figure out the differences . our systems.
if we can determine the smallest import that causes a problem , we may be able to catch an error in the error. For the other thing I can try to actually write to debug output to a file. the pauses you see every she seconds represent the database "cleanup" . I think if I am understanding you it never slows down it just crashes at different points. Let's not try the separate JavaScript yet, but I think the better thing is to give you something with lots of other output. hopefully I can do that tomorrow. sorry it is such a big mystery so far. have to shutdown|

[hfreemanPE]

CST time. just loaded Node.js 12.16.3 and have command prompt for it.

[hfreemanPE]

I just ran 0.1.1 UpdateCycle set to 1 and imported 77767 EML in 82 folders 3.58GB into 12 folders with 3.49GB in 1 hour 56 minutes. It completed without any issues that I can find. There were not any EML in the top directory on this run. It seems like the smaller number on UpdateCycle runs slower but does not crash. That is a better solution as one can let it run overnight if need be for large files. Hopefully this is linear and so say 210k files would run take about 6 hours for example.

On 5/23/2020 10:56 PM, Christopher Leidigh wrote:

@hfreemanPE https://github.com/hfreemanPE I was just about to sign off (I am on the East Coast of the US), I assume you are somewhere ~12 hours difference? I was hoping to be able to post something tonight, but the other things took to long. I was amazed at the speed of the report . test I did today. it does make me more confident that we just have to figure out the differences . our systems. if we can determine the smallest import that causes a problem , we may be able to catch an error in the error. For the other thing I can try to actually write to debug output to a file. the pauses you see every she seconds represent the database "cleanup" . I think if I am understanding you it never slows down it just crashes at different points. Let's not try the separate JavaScript yet, but I think the better thing is to give you something with lots of other output. hopefully I can do that tomorrow. sorry it is such a big mystery so far. have to shutdown|

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thundernest/import-export-tools-ng/issues/85#issuecomment-633175328, or unsubscribe https://github.com/notifications/unsubscribe-auth/APVHJOSXCD4XOZ5J3YYQPNDRTCLINANCNFSM4K3OUFKA.