How to use Apple export script to import to Thunderbird

[johnstonesnow]

I have a load of local folders in Apple Mail (NOTE: It's running on OSX 10.14 Mojave, old style)i. exported these from Mac Mail which took a few hours at least.

I have followed instructions to import into TB via this plugin. Tried every option possible but fails every time with message: "Import Done 0/0"

Any ideas appreciated.

[johnstonesnow]

Not sure I understand. I am going from Mac Mail to Thunderbird?

[cleidigh]

oh right, that might be correct

[johnstonesnow]

Ok, looks like there' may be a problem.

Example:

One folder in particular, on Mac Mail it shows as having 37,374 messages in it. in TB on Fedora the same folder after the import contains 5127 messages.

Same folder on Mac contains emails back to 2006 but in TB only goes back to Feb 2023.

Also, the emails in that folder on TB seem duped:

I wonder if I should start again after runing a "rebuild" of mailbox on Mac. Not sure it can handle doing so, but could try.

[johnstonesnow]

Please don't feel any pressure to respond now. This is not urgent. But I will wait for your thoughts before doing anything else in case you have ideas. Thanks

[cleidigh]

Ok, unfortunately it sounds like we may have to do some mbox debugging. Isolate the folder that has that big discrepancy. We may have to look at it. Also I have beta for v14.0.2 which might have some relevant tweaks.

[johnstonesnow]

Is this something I can do on Fedora if I get that specific mbox folder onto it via USB stick? Or do I need to be on Mac?

"beta for v14.0.2 " - is this a reference to the script or the plugin/TB?

[cleidigh]

we can compare the usb export to the import all on fedora. v14.0.2 is the next ImportExportTools release I am finishing up now. It has a couple of mbox fixes. I am wondering if they may be relevant.

Let's find the folder and deal with it individually.

[johnstonesnow]

ok I have that file/folder in Fedora file manager. I checked on Mac, the exported mbox for that folder reports as 2.15GB on Mac. Same here on Fedora, so the file got here :)

[cleidigh]

Ok now on Thunderbird, right click on the folder and choose the IETNG option open folder location. Then get the file size of the folder file with no extension.

I'm going to go offline in a bit. We may have to take a deeper dive tomorrow. Unfortunately sometimes its very hard to diagnose without seeing the files.

[johnstonesnow]

Sure, no problem at all.

Hmm. Dangit. Tried that many times, nothing happens, FileManager doesn't even open. That could be a Fedora issue though

PS in case it helps, I don't mind if the folders get 'un-nested', would there be any better chance of it working if i just imported one folder at a time, I can shift them around later?

[johnstonesnow]

PPS, drop this whenever you like. Please. I could always delete all, start again and try the new revision of the script whenever it's ready.

[cleidigh]

well I think we look at the individual mbox from the usb and import that file.

[johnstonesnow]

I tried right clicking and choosing Properties on that folder, so I can get the location from there. Also, is the rebuild worth a shot? Assuming not.

[johnstonesnow]

well I think we look at the individual mbox from the usb and import that file.

I see. So I could create a new top level folder in Local Folders and import just that folder, shall I do that?

(Please let me know when you go offline. I don't want my replies to make you feel you have to stay any longer than you want to!)

[cleidigh]

No there is something tripping up a bunch of messages. I'm surprised as I have done a lot of work on this. If you look at the end of the folder do you see messages without a subject or date?

[cleidigh]

yes import separately I'll let you know when I have to go This is important to me that this is solid.

[johnstonesnow]

at the very bottom (which is oldest dated email as i re-sorted with oldest at bottom) there are 3 messages (which represent just one, twice duplicated) and THOSE THREE show no Subject, but do show a date. If that helps?

I will do a manual separate import of just this troublesome folder and let you know if i get a better number of imported emails.

[cleidigh]

ok that means email separators or From lines not being detected correctly. Tomorrow we can try my beta.

[cleidigh]

now I have to go

[johnstonesnow]

thank you, bye for now

[johnstonesnow]

PS I have stopped doing anything this end, I wont do the manual import as I expect to be wiping clean slate for another run. Til next time!

[johnstonesnow]

PPS - Final msg of today, as i fear it may be important to note:

You know how many years (nearly 2 decades!) of emails we are stretching back with my mbox files. All of that time was spent on macs, which were cloned to next machine when i bought newer machines (only a couple in that time). I DO remember having to "rebuild mailboxes" once or twice over the past 10-15 yrs. I haven't done it for a VERY long time though, and wondering if it's just worth considering, IF my old dog of an iMac is capable of it, rebuilding all those mailboxes and exporting again before trying a new conversion.

Just food for thought, ignore me as required! Cheers M

[cleidigh]

@johnstonesnow I think the rebuild makes sense. Import that to a new folder so we can compare to the old. After that we want to find small folders that also have errors. @cleidigh

[johnstonesnow]

heading to desk now to recap on above and see what to do next

[johnstonesnow]

Ok, so a rebuild in Mac Mail, a fresh export, then a fresh conversion.

Damn it I should have done this earlier, a rebuild it could take hours on this machine. I will kick it off now and see what happens. thanks Christoper

[johnstonesnow]

I think I should probably do both a rebuild and reindex, to be as sure as possible of intergrity:

https://www.computerworld.com/article/2476217/mac-os-x-how-to-fix-apple-mail-by-rebuilding-and-reindexing-the-mailbox.html

[cleidigh]

We can do some fun bash stuff on Fedora while it churns. I was thinking about how to analyze this last night. First we should verify the folder count and look at folders skipped, second find the smallest folders with errors. We can look on Fedora for these. Also can we rebuild individual folders?

[johnstonesnow]

You and me have very different ideas of "fun" !!

If you mean on Mac Mail, yes. In fact every guide I have seen says to click on a mailbox (which i think means mail folder) and choose "rebuild". I will be here forever doing that, so I block highlighted all my local folders (all in export tree) and asked it to rebuild all. It may have hung, it started but says "41 mailboxes selected (68586 messages)". We already know there's a lot more messages than that in those folders. Maybe I should kill it

[cleidigh]

maybe on mailbox at a time? But before you do that lets focus on a couple small folders first. Once we id those we can rebuild those quickly. Can you check on some smaller ones?

[johnstonesnow]

I killed it. I can go one at a time yes, but thing is... Mac Mail doesn't give ANY indication of whether it's working or not so I have no clue what its doing (if anything). Even the "Activity" window never shows anything even when it is dong something! Been like it for years, normal though I think.

Ok I am at your disposal. What do you mean by check on smaller folders, in Mac Mail, in Mac Finder (exptree), or in Linux?

[johnstonesnow]

PS that Acivity Window appears to only be for send/receive activity, I just saw it flash up that it was checking for new msgs.

[cleidigh]

well if you can traverse your max first and find a couple small folders then we can check the Thunderbird import of the same to look for a delta in messages. Once we find we can rebuild those and see what happens

[cleidigh]

we can also search in bash terminal

[johnstonesnow]

I have clicked on each and every mailbox on mac mail and asked it to Rebuild.

No response/feedback, but at least I clicked on every single one. Maybe it happens very quickly for LOCAL messages, maybe the time some say to expect it to take is due to most people doing it on IMAP folders where it clears and re-syncs everything over the web.

Are you asking me to look at Mac Mail, or mac Finder? Do you just mean pick out a few folders on Mac Mail which dont have a lot in, and compare those to their equivalent TB folders on Fedora for discrepancies?

[johnstonesnow]

I do wonder if the best thing to do would be to force a reindex of Mac Mail, do you think I should? I am happy to just sit here and do exactly what you suggest, one command at a time if necessary

[cleidigh]

your last statement, look in mac mail to find small folders then we look at Thunderbird

[johnstonesnow]

ok, will do some comparisons now thanks

[johnstonesnow]

ok. Some random picks:

1. Folder called "Legal". On Mac - 102 messages. On TB - 102 messages.
2. Planning - 7 msgs on both
3. Security - 6 msgs on both
4. Software - 13 msgs on both

So far they all seem the same. Weird! But won't hurry, will now go look at some slightly bigger ones....

[johnstonesnow]

5. Privacy - 80 both
6. ok, first discrepancy, but MORE on TB - Folder called H&J - 185 msgs on Mac, 192 on TB. Weird, not sure how to work out what's missing. I guess i can sort them the same way (which they aren't by defaut) and manually compare one by one.

[cleidigh]

good so far, that shows we are normal for those

[cleidigh]

good that will be easier to compare.

[johnstonesnow]

I had a theory which now looks like a failure. I wondered if Mac was counting conversations as 1, where TB counts them as however many msgs within it. But that's not the case. If that were the case, the discrepancy between H&J folders should be 20-30 emails. but it's only 7. That's odd. Shall I manually check every one to try and find the extras on TB?

[cleidigh]

no lets wait and find the smallest mac folder that has fewer messages on Thunderbird

[johnstonesnow]

OK but i have just been manually checking each email, may as well say what happened:

I went down from newest at top in H&J folder on both machines, cross referencing every single msg from top down.

Same until i noticed one appearing to be missing on TB, but it was actually further down, due to having a different DATE! On Mac the message date was August 29 2023 @ 15:20, same email in same folder on TB is dated July 17 2023 @ 10:36.

That's pretty weird! Let me know if you want me to stop comparing and find a folder as you said in your last message, or investigate this H&J folder further

[cleidigh]

yeah I think the folders with missing messages is best start

[cleidigh]

but note for later

[johnstonesnow]

Roger that.. Will find one

[johnstonesnow]

I quickly finished the manual check through H&J, was over half way.
I did find one (so may have missed others) instance of TB having a carbon copy duplicate of the same email, so 2 of same where only 1 of it exists in Mac folder. That may explain the slight increased numbers on soe TB folders.

Now to find a TB folder with fewer msgs...

[johnstonesnow]

Ok. I could describe my folder tree as being in two parts, whilst they are all in the same tree, there is a section at the top which is different. I basically have:

1. Live Folders - where I store all imprtant stuff related to certain topics/people.
2. What I call (and think of as) Archive folders. These are all subnested under a folder called Email Archves, and they contain all my sent items from past email accounts, a trash archive (I know I know!), various other old archives I keep because I'm just weird i guess. :)

Point being:

ALL Folders in Category 1 above seem good, at least numbers wise. There are NO TB folders with FEWER msgs than the Mac Mail counterpart. Some have a few more msgs in on TB (prob just harmless random duplicates) but none have less. That's Good .

Category 2 folders are, by nature, damn big. I wouldn't say it's unimportant if I lose some emails in those, but it's probably less important, whilst nice to have them all.

So I am working up through the archive folders now.

First one is "Trash Archive" - On Mac this shows 11797 msgs. On TB that folder shows 9672 msgs

I will keep checking but give you that bit of info in case you want me to stop here. I MAY find a smaller example but suspect they are all bigger beyond this one in archive folders.

[johnstonesnow]

ok next one is "Sent Archive" and this is bigger, with a bigger problem.

Mac - 37,374 msgs TB - 5116 msgs

(this is the one we looked at yesterday where I noticed the last few messages (most recent dates in folder) has no subject line)