Export with Subfolders (recursive) for EML, HTML, PDF and Plaintext - Testing and Feedback

cleidigh commented 8 months ago

@kiki-ja @tidebre67 @DenB10 @librelegal @warrendodge @pe4000 @zwergneun @Thorsten-Nutzer @projecttocs @grzegorzsurowiec @Yarny0 @Mr-Update

All I have tagged you as a power use, someone I have done a fix or tweak, translators or all the above!

Adding subfolder ( recursive ) export is a way overdue addition. I had to patch it in to code that all has to be rewritten for the next esr. It's pretty important it gets as many eyeballs on it as possible. Thanks for any testing. If you are not interested, no problem, just unsubscribe.

Translators: I have not finished the English so I will post when that is ready.

Notes:

I made a change to the output structure for both single and recursive exports, this is for HTML, PDF and plaintext :

Subfolder export of Inbox:

Inbox_20240225-1330 Inbox index.html messages msg1.html msg2.html Attachments Subfolder1 index.html messages msg1.html

For EML I did it differently so that the structure matches the required directory structure for importing EMLs:

Inbox_20240225-1330 Inbox index.html msg1.eml msg2.eml Subfolder1 index.html msg1.eml

You can import at the top container level and you get the folders created even if you import at the account level.

For PDF with attachments we will have to wait until I do my first rewrite for the next esr using more webextension apis.

Any comments or suggestions welcome.

v14.0.2-b10 https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b10-tb.xpi

Cheers, Christopher

DenB10 commented 8 months ago

@cleidigh

I've noticed a problem when exporting sub-folders in HTML (or PDf or plain text): the "messages" folder is not created in the sub-folder.

Inbox_20240225-1330 Inbox index.html messages msg1.html msg2.html Attachments Subfolder1 index.html messages msg1.html

In this way, messages contained in the subfolder are inaccessible by clicking them in the index (wrong path).

Cheers,

Denis

cleidigh commented 8 months ago

@DenB10 Is there any chance that the option :

create a new directory with an index for messages
on the Export directories tab, is turned off?
Do you get a top container directory?
Can you capture the console? Surprised as I did a million exports... Christopher

DenB10 commented 8 months ago

@cleidigh I've uninstalled then reinstalled IETNG, cleared the cache, the options are at their default values (the option you mention is enabled). I sent myself 3 messages, I put one of them in a sub-folder. I still get the same result when exporting to HTML:

Courrier entrant_date Courrier entrant messages message1.html message2.html sous-dossier message3.html index.html index.html

message3.html is not in a folder "messages".

I'm attaching a copy of the console: console-export-2024-2-27_21-28-0.txt

cleidigh commented 8 months ago

@DenB10 what about messages that's not a folder?

DenB10 commented 8 months ago

@cleidigh "messages" is an explorer folder containing html messages, isn't it???

cleidigh commented 8 months ago

I assume it has the top two messages? So I think I understand it's the second level that lacks the messages directory? C

DenB10 commented 8 months ago

That's right.

cleidigh commented 8 months ago

@DenB10 Thanks. It's hard to do the trees because the indents go away! @cleidigh

DenB10 commented 8 months ago

Thanks. It's hard to do the trees because the indents go away!

In fact, I haven't figured out how to make indents. Sorry. Maybe this file is more comprehensible: FileList.txt

cleidigh commented 8 months ago

@DenB10 You can use the windows tree command tree /f Then paste between three backticks, that gives a nice tree. Christopher

DenB10 commented 8 months ago

@cleidigh Like this?

D:\Users\DenB1\Downloads\Courrier entrant_20240227-2127>tree /f D:.

+---Courrier entrant
    ¦   index.html
    ¦
    +---messages
    ¦       20240227-message1-19959.html
    ¦       20240227-message2-19963.html
    ¦
    +---sous-dossier
            20240227-message3-19971.html
            index.html

Thanks for teaching me something :wink:

cleidigh commented 8 months ago

@DenB10 Great little tool. Should use graphics characters, not sure why yours did not. @cleidigh

pantropia commented 8 months ago

Helo! I would very much like to try this beta version but I can't see how to download it. There's a download button but it doesn't seem to do anything...

cleidigh commented 8 months ago

@pantropia Excellent! I literally just posted b11 with a bug fix and full localization. You just click on the link below and then install from the addon page by file. https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b11-tb.xpi

What formats do you use? @DenB10 This fixes the messages directory bug. @cleidigh

pantropia commented 8 months ago

Ah. Apparently I should have been clicking the Raw button not the Download one. Obvious!

cleidigh commented 8 months ago

@pantropia Were you able to install and run? @cleidigh

DenB10 commented 8 months ago

@cleidigh

This fixes the messages directory bug.

Yes, it works! You even translated the menu items for me, which is wonderful. Life is good. Thanks for everything.

cleidigh commented 8 months ago

@DenB10 Excellent! Yup, Google did the hard work %-) @cleidigh

pantropia commented 8 months ago

@pantropia Were you able to install and run?

Yes it looks like it worked but I'll need to do some faffing about to determine whether it's done it all properly. I'm basically wanting to export everything to .eml files both because I don't like backups I can't see the contents of without restoring them and because I know there's a load of duplicated stuff in various accounts due to a mishap some time ago which I'd like to sort out. I tried EMClient for a while (didn't like how it does rules) but that was able to export all mail from all accounts/folders to a given location in a single operation. I couldn't see any way to export whole accounts to .eml with this, never mind doing all accounts at once, which is a shame, but recursive subfolders is still a considerable improvement over what I would have had to do without it - one old account has folders many layers deep for a selection of mailing lists which had been subdivided into subfolders for year and then month. To know if it worked properly, I'll need to compare the export to what came out of EMClient, but because the structures are different I'll need to either spend some time amending one of the structures to match the other before using a sync tool to compare them or use a duplicate finder (the one I use doesn't do multiple simultaneous sessions and is currently trying to deal with a 3TB folder off my old NAS). Before I start messing around I'll want to look into how the import from .eml files works, as I don't want to end up creating a perfect deduplicated and properly organised copy of my mail which I then can't get back into Thunderbird!

pe4000 commented 8 months ago

Hello Christopher, the function: "Export messages in → plain text format → messages and attachments" no longer works on my computer in version v14.0.2-b10. Greetings, Peter

DenB10 commented 8 months ago

@cleidigh @pe4000

I agree with Peter Indeed, I've just noticed that all the functions for exporting selected messages in both plain text and HTML have been broken since beta 10, i.e. since the successful addition of the recursive export. More work for Christopher, sorry! :cry:

Edit: On further investigation, copying an entire message to the clipboard no longer works either. Something's wrong with exportTools.js

@DenB10

cleidigh commented 8 months ago

@DenB10 @pe4000 First everyone should be on b11

https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b11-tb.xpi

I am not seeing issues with html or plaintext options. Perhaps more details and a console grab? I did reproduce the copyToClipboard issue. No idea how that broke... @cleidigh

DenB10 commented 8 months ago

@cleidigh Sorry but I'm actually using beta 11 and I'm seeing these issues. Console copy:

menu update  
Object { menuIds: (30) […], contexts: (2) […], viewType: undefined, editable: false, pageUrl: undefined, selectedMessages: {…}, displayedFolder: {…} }
wextMenus.js:1140:11
check wextMenus.js:1295:11
tb.ui.interaction.message_display - The key length must be limited to 72 characters.
Object { menuItemId: "msgCtxMenu_Exp_HTMLFormatSaveAttsCreateIndex_Id", parentMenuItemId: "msgCtxMenu_Exp_HTMLFormat_Id", viewType: undefined, editable: false, pageUrl: undefined, selectedMessages: {…}, displayedFolder: {…}, modifiers: [], button: 0 }

Object { id: 1, index: 0, windowId: 3, highlighted: true, active: true, status: "complete", width: 866, height: 357, cookieStoreId: "firefox-default", url: "imap://denb10@imap.free.fr:993/fetch%3EUID%3E/INBOX%3E20036", … }
wextMenus.js:846:11
expMenuDispacher: winId 3 
Object { command: "WXMCMD_HTML_Format", params: {…} }
expMenuDispatcher.js:48:10
expMenuDispacher focused:  true expMenuDispatcher.js:49:10
ChromeWindow chrome://messenger/content/messenger.xhtml
expMenuDispatcher.js:50:10
TypeError: msgFolder is null exportTools.js:1426:2

@DenB10

DenB10 commented 8 months ago

Maybe a screenshot is more readable. Presse-papiers-1

DenB10 commented 8 months ago

@cleidigh

Another issue I just realized: With b9, I could import eml via the context menu of an imap folder, not with b11!!!! The menu item has disappeared.

cleidigh commented 8 months ago

@DenB10 line 1426 is a debug statement, that causes the copyToClipboard to fail, I found that. It makes no sense for msgFolder to be undefined for any export. I will have to add debug I guess. @cleidigh

cleidigh commented 8 months ago

@DenB10 So your output is missing a lot of debug, it doesn't seem to follow the normal path. Did you reboot? @cleidigh

cleidigh commented 8 months ago

@DenB10 Ok I realized you were doing a selection export. I have only been doing folder exports for now. @cleidigh

DenB10 commented 8 months ago

@cleidigh

Sorry for the late reply (I was asleep). I didn't make any selection in the console entries. I just deactivated a very verbose extension and emptied the contents of the console display, performed the HTML (or plain text) message export operation from the email context menu and copied the console contents. Let me know if I need to do anything special (develop an entry, ...).

However, I've just noticed something strange that solves all the problems: if I inactivate this famous line 1426, exporting becomes possible, as well as copying to the clipboard. If that helps... Edit: No, not all: still impossible to import .eml files into an imap folder (no menu item).

@DenB10

cleidigh commented 8 months ago

@DenB10 Yes, the debug line 1426 for subfolders has a msgFolder which is not defined for the selected messages or copyToClipboard so I inadvertently broke those. Easy fix. Will post later. @cleidigh

DenB10 commented 8 months ago

@cleidigh

Since beta 10, I can no longer import .eml files from the context menu of imap folders, as the appropriate menu items no longer appear. It's a function that's very important to me. I've found a workaround by inactivating lines 1304 and 1308 in the wextMenus.js file, but I don't know if it's right.

@DenB10

cleidigh commented 8 months ago

I was fixing another menu issue that disabled the eml option

On Sat, Mar 2, 2024 at 4:04 PM DenB10 @.***> wrote:

@cleidigh https://github.com/cleidigh

Since beta 10, I can no longer import .eml files from the context menu of imap folders, as the appropriate menu items no longer appear. It's a function that's very important to me. I've found a workaround by inactivating lines 1304 and 1308 in the wextMenus.js file, but I don't know if it's right.

@DenB10 https://github.com/DenB10

— Reply to this email directly, view it on GitHub https://github.com/thunderbird/import-export-tools-ng/issues/538#issuecomment-1974909012, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGAZ7625QOEDGMRZRY4L3ATYWI5GZAVCNFSM6AAAAABDZFVZ42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZUHEYDSMBRGI . You are receiving this because you were mentioned.Message ID: @.***>

cleidigh commented 8 months ago

@DenB10 Got everything fixed, will post tomorrow after testing. @cleidigh

DenB10 commented 8 months ago

@cleidigh I did a quick test. All functions seem to work as expected. To be confirmed by other users? @DenB10

cleidigh commented 8 months ago

@DenB10 @pe4000 B12 fixes:

selected mssg export
copyToClipboard
eml imports to imap
various locale fixes
support for old mac CR terminations

https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b12-tb.xpi @cleidigh

DenB10 commented 8 months ago

@cleidigh

Thank you :1st_place_medal:

cleidigh commented 8 months ago

@DenB10 Merci mon ami, c'est tout bien. No more additions so I think I have to do some good regression testing. You have seen how easy it is to break things. Just have to see if user will complete cz locale contribution. Thanks for all The help! Christopher

Yarny0 commented 8 months ago

Tested with

Thunderbird 115.8.0, Linux on x86-64
IET NG v14.0.2-b12

Some changes to IET default settings (likely irrelevant for this test report)

``` author.max_length: 32 autobackup.dir: "/tmp" autobackup.save_mode: 0 clipboard.always_just_text: true csv_separator: "|" export.cut_subject: true export.filename_date_custom_format: "%Y%m%d-%H%M" export.filename_pattern: "%D-%a-%s" export.filenames_toascii: true export.format_warning: false export.help.showOnInstallAndUpdate: false export.import_warning: false export.index_date_custom_format: "%Y-%m-%d" export.mbox.use_mboxext: true exportEML.dir: "/tmp" exportEML.filename_format: 2 exportEML.use_dir: true exportMBOX.dir: "/tmp" exportMBOX.use_dir: true exportMSG.dir: "/tmp" exportMSG.use_dir: true recipients.max_length: 32 subject.max_length: 48 ```

with four accounts

IMAP/maildir (emails also stored locally)
POP3/maildir
POP3/mbox
IMAP/mbox (emails also stored locally)

each with an Inbox with four mails (one with attachemnt) and one subdirectory, again with four emails (one with attachment).

I tested some export function in the context menu of each account's Inbox. These export functions generated results that look reasonable:

single mbox file
single zipped mbox file
mbox, structured with subfolders
mbox, with flattened subfolders
all messages / html / messages & index & attachments with subfolders

Note: I observed that the zip export does not work with IMAP if mails are only stored on the server. However, there is a warning about this, so I don't consider this as a bug.

I am using mainly the EML export functions; these export functions produce exactly what I expect them to produce:

all messages / eml / messages & index
all messages / eml / messages & index with subfolders

Suggestion for improvement: When exporting a directory structure as HTML, a directory's index.html could also list subdirectories and link to the corresponding index.html files, like it lists and links messages. E.g. in

Inbox
+ Subdir
  + index.html  <-- links to msg1 and msg2
  + Messages
    + msg1.html
    + msg2.html
+ index.html  <-- links to msg3 and msg4, could also link <a href="Subdir/index.html">Subdir</a>
+ Messages
  + msg3.html
  + msg4.html

IET is a very valuable tool for me. Thanks for keeping up the development!

DenB10 commented 8 months ago

@Yarny0

First of all, I'm impressed by your rigorous testing and presentation - well done!

Secondly, I had also thought of your suggestion for improvement, but I wonder whether the added value provided is sufficient, bearing in mind that any improvement is likely to break other functions.

But if it's easy to implement, why not? It's up to Christopher to decide!

cleidigh commented 8 months ago

@Yarny0 I second @DenB10 comment, your detailed report is very helpful and things look good. I like your suggestion. It is actually pretty difficult in the current code structure because each folder export is separate and the structure is not understood by any other. However, I have to rewrite all this old code. At that point I will make the index have more of a global view which would facilitate this. @cleidigh

thunderbird / import-export-tools-ng

Export with Subfolders (recursive) for EML, HTML, PDF and Plaintext - Testing and Feedback #538