Feature Request: Make all folders in IMAP folder structure visible to quickMove/quickJump with non-synced folder tree

[jaysmithgit]

(Thunderbird 78.7.1 on Kubuntu Linux 20.04)

We have a small multi-user environment with tens of thousands of mail-containing (on IMAP) folders, in a deep and wide folder/directory structure up to 12 levels deep. We do not download the contents of mail messages to the local user's profile structure.

Currently (and for the last 10+ years), whenever we install a significant Thunderbird upgrade or when a user's Thunderbird panacea.dat file gets corrupted (typically, pre- TB 78, corrupted or became too large every 18-20 months), we have to rebuild the user's profile IMAP structure so that .msf files are created for each folder.

Having the .msf files exist has been the only way that I have found for "filing programs" like QuickFolder or Nostagy/Nostalgy++ to "see" the structure and the folders so that the user can, in such a program, search for the folders into which the user wishes to move or copy emails, or to go-to the searched-for folder.

The only way that I have found in the last 10+ years to build/re-build that structure [user's profile IMAP structure containing the .msf files] is to...

... Manually, using the left-side folder pane in the Thunderbird GUI, open each folder that contains either folders-containing-emails and/or folders-that-thus-contain-more-folders. One I manually open/expose each folder, then Thunderbird creates the .msf file for the folders contained within the folder just opened.

In our environment, prior to TB 78.7.1, we were using a much older TB version and this opening/exposing process could take one person several hours (at least 4-5 hours if one sits in front of the program and waits until they can do each one) for each user/seat. Under TB 78.7.1 and with faster new computers and more memory, it now only takes about 2 hours. However, currently, with 4 users/seats, that is still more than a full day of clicking on folders to open them.

Important: When I say "open/expose", I mean click on any folder that contains folders. It is not necessary to click on the actual mail-containing folders that don't contain folders.

Example of structure: There are far more than 10,000 folders that actually contain mail themselves. Other folders contain either folders-that-do-not-contain-mail themselves and/or folders-that-do-contain-mail. [When I say "contain mail", the mail message content stays on the IMAP server and is not downloaded to the user's computer in our environment.]

user@foo.bar.com [Local User Folders (Inbox, Sent, etc.) ... NOT of concern for this issue] IMAP folder, i.e. public/ ...... folder-level-1-company-abc ...folder-level-2-marketing ......folder-level-3-current-projects ........folder-level-4-project-x ..............folder-level-5-meeting-plans (contains mail) ..............folder-level-5-meeting-proposals (contains mail) ..............folder-level-5-contracts ...................folder level-6-contract-1 (contains mail) ...................folder level-6-contract-2 (contains mail) .........folder-level-4-project-y ..............folder-level-5-meeting-plans (contains mail) ..............folder-level-5-meeting-proposals (contains mail) ...folder-level-2-personnel ..................[lots of structure] ...folder-level2-customers ..................[lots of structure]

For years, I have been operating under the assumption that the .msf files have to be created -- but mail-contents of a folder (if any) does not have to be viewed -- in order for the search functions in programs like QuickFolders to list them in the search function.

It may be now that all that is required is for the folders to be "Subscribed to", but that still requires a lot of time/work, though it is easier than it used to be.

I am open to any possibilities, but based on my previous experience, it seems that what is needed is the a function that walks / iterates through the entire structure and "opens/exposes", in turn, all the folders so as to create .msf files for them all.

[RealRaven2000]

To start I changed the title to be less based on implementation detail (whether they are stored in msf files is a detail which isn't really part of your requirement, and also the Add-on cannot see these files anyway). The front-end code currently works with XPCOM interfaces that are built on top of GLODA (Thunderbird's global database) which is updated through complicated asynchronous code that cannot be changed / overwritten by an Add-on - it's mainly written in C++ and compiled (hard-baked) into the Thunderbird platform. So if it has hard limitations then we will have to find other workarounds (such as forcing the folder tree to update itself by connecting to the server and creating the missing data)

The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree. If a folder has an (even empty) subfolder this is named with the ".sbd" extension, which doesn't necessarily have to contain any files (it can be an empty folder with no messages or it can just contain subfolders).

The "Repair Folder" command which can be initiated from the folder properties dialog will parse the raw message file and build a database index and as such delete and rebuild the .msf file so that contents can be accessed quickly and the Thunderbird search functions can work on it. The process is called "indexing folders" and this activity can be witnessed in Tools / Activity Manager. Large amounts of data / folders will result in long processing times which can be seen there.

For years, I have been operating under the assumption that the .msf files have to be created -- but mail-contents of a folder (if any) does not have to be viewed -- in order for the search functions in programs like QuickFolders to list them in the search function.

This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder. My guess is that it will not generate any subfolders for any unsubscribed folders, so there is no way for an Add-on to see these folders either. The Add-ons cannot talk directly to the Server. A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.

As I was digging around I did find a Server Setting (clicking on the Advanced button) with a checkbox [x] Show only subscribed folders which seems to imply that it should be possible to show all folders without subscription. The hope is to find a Thunderbird function (or set of functions) that updates this information from the remote data into the folder tree.

[jaysmithgit]

Unfortunately, experimenting with what you have described has not resulted in a solution for me.

The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree.

In our environment, in the user's profile IMAP structure such as: /d/home/USER/.thunderbird/12345abc.default-release/ImapMail/mail.FOO.com/public.sbd/...

1) There are ONLY .sbd directories and .msf files. There are NO files or directories without extensions.

2) It does not matter in that regard (no files without extensions) whether or not the folders have or have not been subscribed too.

3) I am not sure how it is accomplished, but NO mail messages (in the IMAP public structure) are actually downloaded to the user's computer or home directory on the server. That is not a problem or concern for us -- it may be a concern for other environments, however.

4) There seem to be different states of the "tree". In the Subscribe dialog one can see the entire tree, even though .msf files have not been yet created. Even though you can see the entire tree in the Subscribe dialog, you cannot see the tree in the left-side folder tree pane of Thunderbird, until you open each folder-that-contains-folders. See more about this below...

This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder.

I do not know either (of course). However....

1) The IMAP (same as above) folders in the tree are not visible in the left-side folder pane until the user manually opens folders-that-contain-folders-or-files, using the GUI "left-side folder pane".

2) The entire IMAP (same as above) folder tree is visible in the Subscribe dialog box. For our large (many thousands of folders) structure, the entire list is populated in about 2-3 seconds.

3) In old (very old?) versions of Thunderbird, the folder tree in the Subscribe dialog box was first presented (in a new installation) in closed (un-expanded) state. However, at some point along the line, and certainly by 78.7.1, the entire IMAP folder tree in the Subscribe dialog box in a new installation is presented to the user in fully expanded / opened state. Also the user can select all folders in the tree (or a range of folder) by clicking on the first desired folder and shift-clicking on the last desired folder. And then the user can click the Subscribe button to subscribe to a range of folders. This process creates .msf files (and the necessary .sbd directories needed to contain the .msf files), but they are just initial skeleton" versions of the files and they are NOT "visible" to either QuickFolders (or Nostalgy++) search.

4) Apparently the skeleton folders created by the Subscribe dialog are not fully registered by the Thunderbird database -- or are not registered in a manner that QuickFolders (or Nostalgy++) is looking for them. Perhaps it would be worth considering how and what QuickFolders is looking for in the Thunderbird database -- maybe there is something more/different that QuickFolders could be looking for in the database to qualify the folders to show up in QuickFolders search results. I say this because as a result of subscribing to folders the .msf files DO exist (but they are just skeletons) -- and also Thunderbird does know the folders exist in the IMAP structure (because they are shown in the Subscribe dialog).

The "Repair Folder" command which can be initiated from the folder properties dialog will parse the raw message file and build a database index and as such delete and rebuild the .msf file so that contents can be accessed quickly and the Thunderbird search functions can work on it. The process is called "indexing folders" and this activity can be witnessed in Tools / Activity Manager. Large amounts of data / folders will result in long processing times which can be seen there.

Unfortunately, the "Repair Folder" feature is not a solution to this issue. In order to use "Repair Folder", one must -- as you say -- use the folder properties dialog. However, if you are using the folder properties dialog that means that you have gone to that folder using the GUI. And if you have gone to the folder using the GUI, then the complete .msf file (not just the skeleton of .msf from subscribing) has already been created by the process of going to the folder. Sorry!

A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.

As stated above...

1) For my installations, no emails are being downloaded locally regardless subscribing or not, so that is not an issue for us. It could be an issue for other installations.

2) But, in my testing, simply subscribing to a folder creates the .msf file (and any necessary .sbd directories), but the .msf file is only an initial skeleton file and thus (for reasons unknown to me) those initial skeleton .msf files (and whatever is happening by that action in the Thunderbird database) does NOT make the folder visible to QuickFolder's search function.

As I was digging around I did find a Server Setting (clicking on the Advanced button) with a checkbox [x] Show only subscribed folders which seems to imply that it should be possible to show all folders without subscription. The hope is to find a Thunderbird function (or set of functions) that updates this information from the remote data into the folder tree.

As you and I had discussed previously (before this issue was posted), I do NOT have "Show only subscribed folders" check-marked. That (not checked -- i.e. show ALL) has been our standard for more than 10 years.

a) For me, all folders are shown in the Subscribe dialog box within 2-3 seconds after opening the dialog box.

b) In a new installation, the Thunderbird left-side-folder-pane shows (for the IMAP structure) only the first-level "public/" folder. Then when the user clicks to opens that, all existing folders (i.e. ALL folders at the second level) become visible (one, by one, by one; you can see them being created/read, one at a time on a slower system) and .msf and .sbd are created. Then when the user clicks to open any of the folders-that-contain-folders, those become visible in the same way, and so forth as you navigate deeper and deeper. It is only when that manual "opening / expanding" is done does "something happen" that makes the folders visible to the Search function of QuickFolders.

My thought about the solution is either... (I fully admit that I have no knowledge about the internal workings of Thunderbird, add-ons, or what the add-ons can and cannot access, etc. However my sense of the situation is that one or both the following "should" be possible...)

1a) First, the user (or an add-on) actually subscribes to all (desired) folders -- even if this has to be manually done by the user (it is not difficult). Thunderbird does create skeleton .msf files (and the necessary .sbd directories) when the user subscribes to folders. Thus there "should" be a "hook" that QuickFolders can use.

1b) Then, determine what QuickFolders is looking at/for in the Thunderbird database to decide what shows up in the QuickFolders search results list. Thunderbird knows what is and is not subscribed to -- and thus there "should" be a way for QuickFolders to look/for at that subscribed-folders information (in addition to QuickFolder's previous method, so no previously-working method becomes broken).

OR

2) Determine what action Thunderbird is taking in the database, when the user, in the left-side-folder-pane, clicks on a folder-that-contains-folders to open/expand such folders. Whatever action Thunderbird is taking at that time, could possibly be replicated by an add-on which would iterate through the structure. While the add-on cannot access the IMAP server itself, as the add-on opens/expands each folder-that-contains-folders, that resulting information should, at that moment, become available to the add-on from the Thunderbird database.

Note: Even if this add-on-driven open/expand folders process is slow and clumsy, if the user can just walk away and let it run overnight, that is a tremendous improvement over the user having to spend HOURS manually clicking on all those folders-that-contain-folders ... for every user's seat.

Note: Perhaps it would be possible for an add-on to create it's own mirror structure of the folders in such a way that it is more efficient for the add on to search in its own structure. Of course such a structure would have to be updated/refreshed to bring it up to date (because of deletions and additions of folders), but that is a process the user could trigger every so often; or perhaps there is data in the Thunderbird database that the add-on could check to see if a refresh is needed (and then the add-on could inform the user that there is a change to the folder structure and a refresh should be done).

[RealRaven2000]

Unfortunately, experimenting with what you have described has not resulted in a solution for me.

The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree.

In our environment, in the user's profile IMAP structure such as: /d/home/USER/.thunderbird/12345abc.default-release/ImapMail/mail.FOO.com/public.sbd/...

1. There are ONLY .sbd directories and .msf files. There are NO files or directories without extensions.

That's because there are no emails in these folders? An old school advice for people who wanted to force rebuilding their databases was to delete msf files manually (while Thunderbird was not running of course). It would then reindex those folders and build fresh msf files. I suspect either there are no emails only subfolders on the server or no email has been downloaded.

2. It does not matter in that regard (no files without extensions) whether or not the folders have or have not been subscribed too.

It really depends on whether there are mails in a folder on the server I think. If there are mails they are not stored in the msf files, that's just metadata; it could well include header data, but it can be rebuilt by downloading the headers from the server.

3. I am not sure how it is accomplished, but NO mail messages (in the IMAP public structure) are actually downloaded to the user's computer or home directory on the server. That is not a problem or concern for us -- it may be a concern for other environments, however.

The only setting with Imap I know is that you can restrict mails to be not downloaded restricted by size or only download the mail headers (the actual mail is then downloaded from the server for viewing when you click on the entry in the list).

4. There seem to be different states of the "tree". In the Subscribe dialog one can see the entire tree, even though .msf files have not been yet created. Even though you can see the entire tree in the Subscribe dialog, you cannot see the tree in the left-side folder tree pane of Thunderbird, until you open each folder-that-contains-folders. See more about this below...

Yes. My example code tries to do that for you, but it's not yet fully successful and my cause all mail to be downloaded but not exactly 100% of the folders to be downloaded, that's why I didn't upload it here yet.

This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder.

I do not know either (of course). However....

2. The entire IMAP (same as above) folder tree is visible in the Subscribe dialog box. For our large (many thousands of folders) structure, the entire list is populated in about 2-3 seconds.... ... by 78.7.1, the entire IMAP folder tree in the Subscribe dialog box in a new installation is presented to the user in fully expanded / opened state. Also the user can select all folders in the tree (or a range of folder) by clicking on the first desired folder and shift-clicking on the last desired folder. And then the user can click the Subscribe button to subscribe to a range of folders. This process creates .msf files (and the necessary .sbd directories needed to contain the .msf files), but they are just initial skeleton" versions of the files and they are NOT "visible" to either QuickFolders (or Nostalgy++) search.

Looks like a Thunderbird bug to me.

4. Apparently the skeleton folders created by the Subscribe dialog are not fully registered by the Thunderbird database -- or are not registered in a manner that QuickFolders (or Nostalgy++) is looking for them. **Perhaps it would be worth considering how and what QuickFolders is looking for in the Thunderbird database -- maybe there is something more/different that QuickFolders could be looking for in the database to qualify the folders to show up in QuickFolders search results.

I am using the low level XPCOM interfaces nsIMsgFolder and nsIMsgAccountManager.allFolders for iterating but they will eventually be replaced by even higher level API functions. They abstract away part of the Imap complexities and return a simple Array of items.

Unfortunately, the "Repair Folder" feature is not a solution to this issue. In order to use "Repair Folder", one must -- as you say -- use the folder properties dialog. However, if you are using the folder properties dialog that means that you have gone to that folder using the GUI. And if you have gone to the folder using the GUI, then the complete .msf file (not just the skeleton of .msf from subscribing) has already been created by the process of going to the folder. Sorry!

I did not suggest it as a solution, I merely explained that it rebuilds the metadata in the msf file. I don't think the msf files are necessary for Thunderbird to "see" the folders, but they have to exist on disk. Your problem seems to be that the folders are not generated automatically?

A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.

As stated above...

1. For my installations, no emails are being downloaded locally regardless subscribing or not, so that is not an issue for us. It could be an issue for other installations.
2. But, in my testing, simply subscribing to a folder creates the .msf file (and any necessary .sbd directories), but the .msf file is only an initial skeleton file and thus (for reasons unknown to me) those initial skeleton .msf files (and whatever is happening by that action in the Thunderbird database) does NOT make the folder visible to QuickFolder's search function.

Looks like a bug in Thunderbird, I will ask the thunderbird developers core team why empty folders would be invisible to Thunderbird.

a) For me, all folders are shown in the Subscribe dialog box within 2-3 seconds after opening the dialog box.

The subscribe dialog is obviously not directly connected to the folder tree. That Thunderbird doesn't rebuild the tree looks like a Thunderbird bug to me, but it may be by design. Certainly not something QuickFolders can fix, that's why I suggested the helper function that forces visiting every folder and expanding it so you don't have to do it manually.

b) In a new installation, the Thunderbird left-side-folder-pane shows (for the IMAP structure) only the first-level "public/" folder. Then when the user clicks to opens that, all existing folders (i.e. ALL folders at the second level) become visible (one, by one, by one; you can see them being created/read, one at a time on a slower system) and .msf and .sbd are created. Then when the user clicks to open any of the folders-that-contain-folders, those become visible in the same way, and so forth as you navigate deeper and deeper. It is only when that manual "opening / expanding" is done does "something happen" that makes the folders visible to the Search function of QuickFolders.

And I bet there is a reason for precisely large profiles with many folders or slow connections, since the JavaScript was still single threaded I could imagine starting up Thunderbird and rebuilding the whole tree every time (Imap works more or less in real time) would severely delay it. But that's just a speculation, who knows why the design stuff in a certain way?

My thought about the solution is either... (I fully admit that I have no knowledge about the internal workings of Thunderbird, add-ons, or what the add-ons can and cannot access, etc. However my sense of the situation is that one or both the following "should" be possible...)

1a) First, the user (or an add-on) actually subscribes to all (desired) folders -- even if this has to be manually done by the user (it is not difficult). Thunderbird does create skeleton .msf files (and the necessary .sbd directories) when the user subscribes to folders. Thus there "should" be a "hook" that QuickFolders can use.

We have no direct access to the msf files in the future. only the API methods. At the moment I can still use XPCOM, which the frontend code of Thunderbird also uses, but they are eventually going to take this away. So it is important that Thunderbird must support the building of the folder tree through some command or mechanism we still need to discover. I already did debug the subscribe dialog last Monday for an hour but it is not easy to understand because it is event driven and the C++ parts that run in the background are hidden to me.

1b) Then, determine what QuickFolders is looking at/for in the Thunderbird database to decide what shows up in the QuickFolders search results list.

Even though Thunderbird uses a database internally we have no direct access here.

Thunderbird knows what is and is not subscribed to -- and thus there "should" be a way for QuickFolders to look/for at that subscribed-folders information (in addition to QuickFolder's previous method, so no previously-working method becomes broken). OR

1. Determine what action Thunderbird is taking in the database, when the user, in the left-side-folder-pane, clicks on a folder-that-contains-folders to open/expand such folders. Whatever action Thunderbird is taking at that time, could possibly be replicated by an add-on which would iterate through the structure. While the add-on cannot access the IMAP server itself, as the add-on opens/expands each folder-that-contains-folders, that resulting information should, at that moment, become available to the add-on from the Thunderbird database.

I propose that Thunderbird should be able to build / refresh the tree from some command that's what we need to find. But it may well be event driven such as "we don't bother downloading deeply nested folders until the user expands a node on the tree, which will save time and make the UI more responsive". They only recently started introducing multi-process browsing (fissure) and then the speed penalty of Thunderbird rebuilding a tree in the background would be less punitive for the user.

Note: Even if this add-on-driven open/expand folders process is slow and clumsy, if the user can just walk away and let it run overnight, that is a tremendous improvement over the user having to spend HOURS manually clicking on all those folders-that-contain-folders ... for every user's seat.

I have already written the function but it didn't always reliable build all folders which is annoying. I will probably re-ask the question on the Tb developer matrix chat next week but I cannot do that too often because they will just ignore me as they have a lot of their own code to worry about. I don't even need to ask on the support and Add-on developers forums because that information is so incredibly low-level that even Core developers do not have it in their head - they would have to look up the flow in the code and or maybe try and find a developer who worked on this code over the last 10 years.

Note: Perhaps it would be possible for an add-on to create it's own mirror structure of the folders in such a way that it is more efficient for the add on to search in its own structure. Of course such a structure would have to be updated/refreshed to bring it up to date (because of deletions and additions of folders), but that is a process the user could trigger every so often; or perhaps there is data in the Thunderbird database that the add-on could check to see if a refresh is needed (and then the add-on could inform the user that there is a change to the folder structure and a refresh should be done).

I am not going to do that. It's hard enough synchronizing the tabs with their folders (which users can rename and move) without losing track and I have 10,000 lines of code to rewrite to make QuickFolders compatible as a web extension.

Here is a test version - you get the command to refresh the tree by holding down Shift while clicking the settings menu. Then select Debug > Force Tree Refresh

This will iterate all folders known to the tree at the time and brute force expand them in the tree, the hope is that it will expand all contained folders from the remote server. I did a few tests with my gmail account and closed the parent folder in the tree before I added children and grandchildren, but I am not sure if it is fully reliable. Also at the moment in the end all folders will be expanded in the tree, it's hard to remember which ones should be closed again (and difficult to predict whether that would impact on expanding). To get additional debug information you can enable "debug mode" on the QuickFolders Advanced Settings tab:

QuickFolders-wx-5.6pre30.zip

To install, download the zip file. then drag the zip file into Thunderbird Add-ons Manager. github doesn't allow xpi files in comments.

[jaysmithgit]

Unfortunately, experimenting with what you have described has not resulted in a solution for me.

The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree.

In our environment, in the user's profile IMAP structure such as: /d/home/USER/.thunderbird/12345abc.default-release/ImapMail/mail.FOO.com/public.sbd/...

1. There are ONLY .sbd directories and .msf files. There are NO files or directories without extensions.

That's because there are no emails in these folders? An old school advice for people who wanted to force rebuilding their databases was to delete msf files manually (while Thunderbird was not running of course). It would then reindex those folders and build fresh msf files. I suspect either there are no emails only subfolders on the server or no email has been downloaded.

I think we are having a communication problem, probably because we are each accustomed to different environments??? There are email messages in all the folders (on the IMAP server!!) that are "email containing". Sometimes only one or two email messages and sometimes 10,000 in one folder. However in each user's profile, if the folder tree has been expanded/opened so that the .msf files have been created, there are only .msf files and .sbd directories and NO files that do not have either .msf or .sbd on them. If you use a text editor to view the contents of those .msf files, the contents correctly describe the summaries of the email messages that they represent and for which the complete messages are on the IMAP server.

Our Thunderbird is configured to NOT download messages to the local user's profile space or hard disk. This is intentional. In a multi-user environment, where users are ALWAYS "ONLINE" (with the IMAP server), the data (email messages) should only be stored on the IMAP server.

1. It does not matter in that regard (no files without extensions) whether or not the folders have or have not been subscribed too.

We have NO files (folders) in the user's profile area that do not have extensions.

It really depends on whether there are mails in a folder on the server I think. If there are mails they are not stored in the msf files, that's just metadata; it could well include header data, but it can be rebuilt by downloading the headers from the server.

There are email folders, containing email messages, physically on the IMAP server. None locally in the user's profile space or hard disk. The local user only has summaries (.msf and .sdb).

1. I am not sure how it is accomplished, but NO mail messages (in the IMAP public structure) are actually downloaded to the user's computer or home directory on the server. That is not a problem or concern for us -- it may be a concern for other environments, however.

The only setting with Imap I know is that you can restrict mails to be not downloaded restricted by size or only download the mail headers (the actual mail is then downloaded from the server for viewing when you click on the entry in the list).

Correct. And that is what is happening in our environment.

1. There seem to be different states of the "tree". In the Subscribe dialog one can see the entire tree, even though .msf files have not been yet created. Even though you can see the entire tree in the Subscribe dialog, you cannot see the tree in the left-side folder tree pane of Thunderbird, until you open each folder-that-contains-folders. See more about this below...

Yes. My example code tries to do that for you, but it's not yet fully successful and my cause all mail to be downloaded but not exactly 100% of the folders to be downloaded, that's why I didn't upload it here yet.

I really don't think that downloading of actual mail messages is a serious risk. I would be testing using a test user and I can simply delete that test user's Thunderbird profile directory.

This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder.

I do not know either (of course). However....

1. The entire IMAP (same as above) folder tree is visible in the Subscribe dialog box. For our large (many thousands of folders) structure, the entire list is populated in about 2-3 seconds.... ... by 78.7.1, the entire IMAP folder tree in the Subscribe dialog box in a new installation is presented to the user in fully expanded / opened state. Also the user can select all folders in the tree (or a range of folder) by clicking on the first desired folder and shift-clicking on the last desired folder. And then the user can click the Subscribe button to subscribe to a range of folders. This process creates .msf files (and the necessary .sbd directories needed to contain the .msf files), but they are just initial skeleton" versions of the files and they are NOT "visible" to either QuickFolders (or Nostalgy++) search.

Looks like a Thunderbird bug to me.

1. Apparently the skeleton folders created by the Subscribe dialog are not fully registered by the Thunderbird database -- or are not registered in a manner that QuickFolders (or Nostalgy++) is looking for them. **Perhaps it would be worth considering how and what QuickFolders is looking for in the Thunderbird database -- maybe there is something more/different that QuickFolders could be looking for in the database to qualify the folders to show up in QuickFolders search results.

I am using the low level XPCOM interfaces nsIMsgFolder and nsIMsgAccountManager.allFolders for iterating but they will eventually be replaced by even higher level API functions. They abstract away part of the Imap complexities and return a simple Array of items.

Unfortunately, the "Repair Folder" feature is not a solution to this issue. In order to use "Repair Folder", one must -- as you say -- use the folder properties dialog. However, if you are using the folder properties dialog that means that you have gone to that folder using the GUI. And if you have gone to the folder using the GUI, then the complete .msf file (not just the skeleton of .msf from subscribing) has already been created by the process of going to the folder. Sorry!

I did not suggest it as a solution, I merely explained that it rebuilds the metadata in the msf file. I don't think the msf files are necessary for Thunderbird to "see" the folders, but they have to exist on disk. Your problem seems to be that the folders are not generated automatically?

The .msf files and .sbd directories are NOT -- and never have been in 10+ years -- generated automatically for the folders that are on the IMAP server.

If I "subscribe" to folders on the IMAP server (which I can do), Thunderbird DOES create .msf files skeleton files only and the necessary .sbd directories that are needed. However, QuickFolders (and Nostalgy++) still don't "see" those folders when doing searches.

The only way I know of for QuickFolders (or Nostalgy++) to "see" the folders when searching is if I manually, in the left-side-folder-pane expand/open the folder structure to expose all folders. (I do NOT have to click on terminal [mail-containing] folders; I just have to expand the folder that contains them so that they become exposed/visible.)

A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.

As stated above...

1. For my installations, no emails are being downloaded locally regardless subscribing or not, so that is not an issue for us. It could be an issue for other installations.
2. But, in my testing, simply subscribing to a folder creates the .msf file (and any necessary .sbd directories), but the .msf file is only an initial skeleton file and thus (for reasons unknown to me) those initial skeleton .msf files (and whatever is happening by that action in the Thunderbird database) does NOT make the folder visible to QuickFolder's search function.

Looks like a bug in Thunderbird, I will ask the thunderbird developers core team why empty folders would be invisible to Thunderbird.

The Susbcribe-created .msf folders are NOT "empty" -- they just have a skeleton contents that is essentially only the name of the folder plus about 20 header lines that are common to all folders, but NOT including the summary of the contents of the folder that is on the IMAP server. Would you like me to post an example??

a) For me, all folders are shown in the Subscribe dialog box within 2-3 seconds after opening the dialog box.

The subscribe dialog is obviously not directly connected to the folder tree. That Thunderbird doesn't rebuild the tree looks like a Thunderbird bug to me, but it may be by design. Certainly not something QuickFolders can fix, that's why I suggested the helper function that forces visiting every folder and expanding it so you don't have to do it manually.

It depends upon what you mean by "tree". The Subscribe dialog correctly represents the "tree" on the IMAP server. However, the left-side-folder-pane of Thunderbird does not even know that "tree" exist until the user manually expands/opens the folders-that-contain-folders.

I agree that it is a Thunderbird bug/feature -- but it seems like bad design to me!

The "helper function" sounds like a wonderful solution as you described.

b) In a new installation, the Thunderbird left-side-folder-pane shows (for the IMAP structure) only the first-level "public/" folder. Then when the user clicks to opens that, all existing folders (i.e. ALL folders at the second level) become visible (one, by one, by one; you can see them being created/read, one at a time on a slower system) and .msf and .sbd are created. Then when the user clicks to open any of the folders-that-contain-folders, those become visible in the same way, and so forth as you navigate deeper and deeper. It is only when that manual "opening / expanding" is done does "something happen" that makes the folders visible to the Search function of QuickFolders.

And I bet there is a reason for precisely large profiles with many folders or slow connections, since the JavaScript was still single threaded I could imagine starting up Thunderbird and rebuilding the whole tree every time (Imap works more or less in real time) would severely delay it. But that's just a speculation, who knows why the design stuff in a certain way?

I think you are correct. Our 10-year-old server and gigabit network CAN handle the load -- at least if only done from time to time. However, our PREVIOUS Thunderbird (on older workstations, but the older workstations still had more power than the old server), would not have been able to handle that. I don't think the problem was hardware -- I think that OLD Thunderbird was not up to that task.

My thought about the solution is either... (I fully admit that I have no knowledge about the internal workings of Thunderbird, add-ons, or what the add-ons can and cannot access, etc. However my sense of the situation is that one or both the following "should" be possible...) 1a) First, the user (or an add-on) actually subscribes to all (desired) folders -- even if this has to be manually done by the user (it is not difficult). Thunderbird does create skeleton .msf files (and the necessary .sbd directories) when the user subscribes to folders. Thus there "should" be a "hook" that QuickFolders can use.

We have no direct access to the msf files in the future. only the API methods. At the moment I can still use XPCOM, which the frontend code of Thunderbird also uses, but they are eventually going to take this away. So it is important that Thunderbird must support the building of the folder tree through some command or mechanism we still need to discover. I already did debug the subscribe dialog last Monday for an hour but it is not easy to understand because it is event driven and the C++ parts that run in the background are hidden to me.

Understood. Everything is movng to the API methods. But that means the Thunderbird developers have to do more/better than they have been doing. If they restrict only to API methods, they must also accept the responsibility for annoying issues like this one!

1b) Then, determine what QuickFolders is looking at/for in the Thunderbird database to decide what shows up in the QuickFolders search results list.

Even though Thunderbird uses a database internally we have no direct access here.

My apologies. I thought you earlier said that you did have access to the Thunderbird database.

Thunderbird knows what is and is not subscribed to -- and thus there "should" be a way for QuickFolders to look/for at that subscribed-folders information (in addition to QuickFolder's previous method, so no previously-working method becomes broken). OR

1. Determine what action Thunderbird is taking in the database, when the user, in the left-side-folder-pane, clicks on a folder-that-contains-folders to open/expand such folders. Whatever action Thunderbird is taking at that time, could possibly be replicated by an add-on which would iterate through the structure. While the add-on cannot access the IMAP server itself, as the add-on opens/expands each folder-that-contains-folders, that resulting information should, at that moment, become available to the add-on from the Thunderbird database.

I propose that Thunderbird should be able to build / refresh the tree from some command that's what we need to find. But it may well be event driven such as "we don't bother downloading deeply nested folders until the user expands a node on the tree, which will save time and make the UI more responsive". They only recently started introducing multi-process browsing (fissure) and then the speed penalty of Thunderbird rebuilding a tree in the background would be less punitive for the user.

I agree with you on what is needed.

I agree that it may indeed be "event driven" and only through that left-side-folder-pane.

Note: Even if this add-on-driven open/expand folders process is slow and clumsy, if the user can just walk away and let it run overnight, that is a tremendous improvement over the user having to spend HOURS manually clicking on all those folders-that-contain-folders ... for every user's seat.

I have already written the function but it didn't always reliable build all folders which is annoying. I will probably re-ask the question on the Tb developer matrix chat next week but I cannot do that too often because they will just ignore me as they have a lot of their own code to worry about. I don't even need to ask on the support and Add-on developers forums because that information is so incredibly low-level that even Core developers do not have it in their head - they would have to look up the flow in the code and or maybe try and find a developer who worked on this code over the last 10 years.

Understood. You don't want to annoy the TB developers. :-)

Note: Perhaps it would be possible for an add-on to create it's own mirror structure of the folders in such a way that it is more efficient for the add on to search in its own structure. Of course such a structure would have to be updated/refreshed to bring it up to date (because of deletions and additions of folders), but that is a process the user could trigger every so often; or perhaps there is data in the Thunderbird database that the add-on could check to see if a refresh is needed (and then the add-on could inform the user that there is a change to the folder structure and a refresh should be done).

I am not going to do that. It's hard enough synchronizing the tabs with their folders (which users can rename and move) without losing track and I have 10,000 lines of code to rewrite to make QuickFolders compatible as a web extension.

Understood.

Here is a test version - you get the command to refresh the tree by holding down Shift while clicking the settings menu. Then select Debug > Force Tree Refresh

This will iterate all folders known to the tree at the time and brute force expand them in the tree, the hope is that it will expand all contained folders from the remote server. I did a few tests with my gmail account and closed the parent folder in the tree before I added children and grandchildren, but I am not sure if it is fully reliable. Also at the moment in the end all folders will be expanded in the tree, it's hard to remember which ones should be closed again (and difficult to predict whether that would impact on expanding). To get additional debug information you can enable "debug mode" on the QuickFolders Advanced Settings tab:

QuickFolders-wx-5.6pre30.zip

To install, download the zip file. then drag the zip file into Thunderbird Add-ons Manager. github doesn't allow xpi files in comments.

I will work on testing this over the next couple of days. I will be using a testing user login, so no harm can be done. I will also do it at night when a spike in network load won't inconvenience anybody.

Unless you have already thought of it, one issue that I can see could be a problem is that the process may need to be "throttled" so that it only does X quantity per second or something like that. Otherwise, I can imagine "too many simultaneous processes" or "too many simultaneous hits to the IMAP server". If this helper is just forcing Thunderbird to do certain tasks, one could potentially over-run the capability to keep track of (remember) the tasks being requested -- maybe that could be the cause of the irregular results you noticed in your testing.

I sure appreciate the serious attention that you are giving this issue.

[RealRaven2000]

I am replying (and formatting) directly on github. I cannot handle a message that is longer than 100 lines long. So I am replying from memory. You refer to msf as "skeletons" - they are meta-data and contain the info contained in the mail headers. And the can be rebuild by querying the server. But they have nothing to do with the folders as such so I don't why we keep talking about them. If you delete all your msf files in your IMap folder you will still see the same folders in the Thunderbird folder tree; it is not going to make folders disappear.

You where asking about the "tree". I am referring to the folders that are accessible on the left hand side of the 3pane window, which is called "Folder Tree". These folders seem to correspond with the ones that are accessible to the "allFolders" iterator.

Of course the tree just attempts to represent the real folder hierarchy on the remote server (and the subscribe dialog is able to pull that whole information in a different, less bureaucratic way - probably using a tenth or a hundreth of the memory and event listeners that are generated in the "folder tree" (see above) displayed in the Thunderbird main window.

[jaysmithgit]

I am replying (and formatting) directly on github. I cannot handle a message that is longer than 100 lines long. So I am replying from memory.

Since you told me to use GitHub, I have only replied/formatted on GitHub. I have NOT replied by email.

You refer to msf as "skeletons" - they are meta-data and contain the info contained in the mail headers. And the can be rebuild by querying the server. But they have nothing to do with the folders as such so I don't why we keep talking about them. If you delete all your msf files in your IMap folder you will still see the same folders in the Thunderbird folder tree; it is not going to make folders disappear.

We are having two types of communication problems:

1) Terminology: we are using different words for the same or different things.

2) There are factual differences in our experiences.

We were talking about .msf files because...

A) .msf files. There are TWO different forms of .msf files. They are both .msf files, but how they have been created results in TWO different forms of them. I do NOT know if it is the form of .msf file that makes a difference in my issue OR if it is some OTHER aspect of the different process that creates the two forms of .msf files that cause the difference in my issue.

Aa) .msf files created by using the Subscribe dialog box. This creates .msf files (and the needed .sbd directories), but FOR ME they are only "skeletons". FOR ME they only have the INITIAL header of the .msf file itself -- they do NOT contain any mail-specific data. FOR ME, the process that created the subscribe-created .msf files, or perhaps the presence of these files, does NOT allow QuickFolders to "see" the corresponding folder structure represented by those .msf files. Below I will paste in an example of a Subscribe-created .msf file -- you will see that it contains no mail-specific data.

Ab) .msf files (and the needed .sbd directories) created by (or updated by) manually opening/expanding the folder structure in the left-side-folder-pane. FOR ME these .msf files DO contain mail-specific data such as information from the mail message headers, etc. FOR ME the process that creates THESE .mfs files, or perhaps the presence of these files, DOES allow QuickFolders to "see" the folder structure that THESE .msf files represent.

You where asking about the "tree". I am referring to the folders that are accessible on the left hand side of the 3pane window, which is called "Folder Tree". These folders seem to correspond with the ones that are accessible to the "allFolders" iterator.

Okay. There is also a "tree" in the Subscribe dialog.

The tree in the Subscribe dialog DOES correctly represent the tree that on the IMAP server.

The tree in the left-folder-pane does NOT correctly represent the tree that is either in the Subscribe dialog on the IMAP server. The tree in the left-folder-pane is MISSING 99.9% of the folder structure until each folder-that-contains-folders is manually opened/expanded.

Of course the tree just attempts to represent the real folder hierarchy on the remote server (and the subscribe dialog is able to pull that whole information in a different, less bureaucratic way - probably using a tenth or a hundreth of the memory and event listeners that are generated in the "folder tree" (see above) displayed in the Thunderbird main window.

Understood.

The problem seems to be that Thunderbird requires those "event listeners" to be triggered by manually opening/expanding, in the left-side-folder-pane, all the folders-that-contain-folders.... before QuickFolders can "see" them.

We just don't know why this is true and why programs such as QuickFolders CAN see the the folder structure resulting from the left-pane actions, but CANNOT see the folder structure when subscribing is done using the Subscribe dialog.

Hopefully the test version you posted will give us some useful answers. :-) I have already downloaded it.

I will be testing it over the next couple of days. (I will not be able to work on this tomorrow.) I will report back when I have some results.

=-=-= SAMPLE of an ENTIRE .msf "skeleton" file that was created by subscribing using the Subscribe dialog:

This is all that is in this "skeleton" .msf file (created by subscribing). There is no mail-specific data.

From this location: (obfuscated)

USER@SERVER:/PATH-TO/USER/.thunderbird/ABCD1234.default-release/ImapMail/mail.FOO.com/public.sbd/__jsa.sbd/___File_by_Person.sbd/a-f.sbd/a.sbd

COMMAND LINE: more sensitive_name_of_mail_folder (obfuscated)

// < <(a=c)> // (f=iso-8859-1) (80=ns:msg:db:row:scope:msgs:all)(81=subject)(82=sender)(83=message-id) (84=references)(85=recipients)(86=date)(87=size)(88=flags)(89=priority) (8A=label)(8B=statusOfset)(8C=numLines)(8D=ccList)(8E=bccList) (8F=msgThreadId)(90=threadId)(91=threadFlags)(92=threadNewestMsgDate) (93=children)(94=unreadChildren)(95=threadSubject)(96=msgCharSet) (97=ns:msg:db:table:kind:msgs)(98=ns:msg:db:table:kind:thread) (99=ns:msg:db:table:kind:allthreads) (9A=ns:msg:db:row:scope:threads:all)(9B=threadParent)(9C=threadRoot) (9D=msgOffset)(9E=offlineMsgSize) (9F=ns:msg:db:row:scope:dbfolderinfo:all) (A0=ns:msg:db:table:kind:dbfolderinfo)(A1=numMsgs)(A2=numNewMsgs) (A3=folderSize)(A4=expungedBytes)(A5=folderDate)(A6=highWaterKey) (A7=mailboxName)(A8=UIDValidity)(A9=totPendingMsgs) (AA=unreadPendingMsgs)(AB=expiredMark)(AC=version)(AD=forceReparse) (AE=fixedBadRefThreading)(AF=onlineName)(B0=folderName) (B1=highestModSeq)(B2=highestRecordedUID)> {1:^80 {(k^97:c)(s=9)} } {FFFFFFFD:^9A {(k^99:c)(s=9)} }

<(80=1)(81=0)(82=public/__jsa/___File_by_Person/a-f/a/sensitive_name_of_mail_folder)> {1:^9F {(k^A0:c)(s=9u)} [1(^AC=1)(^AD=0)(^AE=1)(^AF^82)]}

=-=-= END OF SAMPLE

[RealRaven2000]

We were talking about .msf files because...

A) .msf files. There are TWO different forms of .msf files. They are both .msf files, but how they have been created results in TWO different forms of them. I do NOT know if it is the form of .msf file that makes a difference in my issue OR if it is some OTHER aspect of the different process that creates the two forms of .msf files that cause the difference in my issue.

Aa) .msf files created by using the Subscribe dialog box. This creates .msf files (and the needed .sbd directories), but FOR ME they are only "skeletons". FOR ME they only have the INITIAL header of the .msf file itself -- they do NOT contain any mail-specific data. FOR ME, the process that created the subscribe-created .msf files, or perhaps the presence of these files, does NOT allow QuickFolders to "see" the corresponding folder structure represented by those .msf files. Below I will paste in an example of a Subscribe-created .msf file -- you will see that it contains no mail-specific data.

Ab) .msf files (and the needed .sbd directories) created by (or updated by) manually opening/expanding the folder structure in the left-side-folder-pane. FOR ME these .msf files DO contain mail-specific data such as information from the mail message headers, etc. FOR ME the process that creates THESE .mfs files, or perhaps the presence of these files, DOES allow QuickFolders to "see" the folder structure that THESE .msf files represent.

I don't think the msf files create folders or cause folders to be created it's the other way round. Also I cannot access msf files so they are irrelevant to me.

The tree in the Subscribe dialog DOES correctly represent the tree that on the IMAP server.

The tree in the left-folder-pane does NOT correctly represent the tree that is either in the Subscribe dialog on the IMAP server. The tree in the left-folder-pane is MISSING 99.9% of the folder structure until each folder-that-contains-folders is manually opened/expanded.

I know.

I recommend wrapping code etc with the code tags, using the button:

//
< <(a=c)> // (f=iso-8859-1)
(80=ns:msg:db:row:scope:msgs:all)(81=subject)(82=sender)(83=message-id)
(84=references)(85=recipients)(86=date)(87=size)(88=flags)(89=priority)
(8A=label)(8B=statusOfset)(8C=numLines)(8D=ccList)(8E=bccList)
(8F=msgThreadId)(90=threadId)(91=threadFlags)(92=threadNewestMsgDate)
(93=children)(94=unreadChildren)(95=threadSubject)(96=msgCharSet)
(97=ns:msg:db:table:kind:msgs)(98=ns:msg:db:table:kind:thread)
(99=ns:msg:db:table:kind:allthreads)
(9A=ns:msg:db:row:scope:threads:all)(9B=threadParent)(9C=threadRoot)
(9D=msgOffset)(9E=offlineMsgSize)
(9F=ns:msg:db:row:scope:dbfolderinfo:all)
(A0=ns:msg:db:table:kind:dbfolderinfo)(A1=numMsgs)(A2=numNewMsgs)
(A3=folderSize)(A4=expungedBytes)(A5=folderDate)(A6=highWaterKey)
(A7=mailboxName)(A8=UIDValidity)(A9=totPendingMsgs)
(AA=unreadPendingMsgs)(AB=expiredMark)(AC=version)(AD=forceReparse)
(AE=fixedBadRefThreading)(AF=onlineName)(B0=folderName)
(B1=highestModSeq)(B2=highestRecordedUID)>
{1:^80 {(k^97:c)(s=9)} }
{FFFFFFFD:^9A {(k^99:c)(s=9)} }

<(80=1)(81=0)(82=public/__jsa/___File_by_Person/a-f/a/sensitive_name_of_mail_folder)>
{1:^9F {(k^A0:c)(s=9u)}
[1(^AC=1)(^AD=0)(^AE=1)(^AF^82)]}

=-=-= END OF SAMPLE

[jaysmithgit]

I don't think the msf files create folders or cause folders to be created it's the other way round.

You are, of course, correct.

The presence of folders (on the IMAP server) results/allows the .msf files to be created. I thought that is what I had said, but perhaps I did not say it clearly.

.msf files are only created when either:

1) The user subscribes (FOR ME that only creates "skeleton" .msf files and the Subscribe process does NOT result in QuickFolders being able to see those subscribed folders in the tree.)

2) The user manually opens/exposes folders in the left-side-folder-pane tree. Until that open/expand action is taken, the .msf files don't exist (unless previously subscribed, but those are "skeletons"). That open/expand action must, I assume, read the IMAP server and thus creates/updates the .msf file.

From what you are saying, since you say QuickFolders does not do anything with the .msf files... the process of opening/expanding in the left-folder-pane, in addition to creating the .msf files, must do something else in the Thunderbird database (?) that does allow QuickFolders to see those folders in the folder tree. Until that action (opening/expanding) is taken, the Thunderbird database apparently does not know about the existence of the folders on the IMAP server (even though the Subscribe dialog does know about the folders on the IMAP server) -- thus QuickFolders is not able to know about the folders.

I suspect that if the Thunderbird developers were to "reinvent" these processes today, they would do it differently. :-)

I will be testing the "helper function" in the test version you supplied.

[RealRaven2000]

Here is a new version, which tries to restore all the folders that should be collapsed / invisible after the whole sync operation - this will prevent all items in the Imap tree to be expanded at the very end. Hopefully should make testing a little easier as well:

QuickFolders-wx-5.6pre36.zip

Simple tests look encouraging so far. I added some (empty) folders on the Imap server with the parent collapsed in the local tree. Once it was finished I could jump to the new folders using quickMove immediately, even they were collapsed in the final tree.

[RealRaven2000]

And another version - I decided to take out the routine that counts subfolders (I hoped to catch an instance when that changed but apparently it wasn't happening / helpful) which should make it perform faster:

QuickFolders-wx-5.6pre38.zip

One thing it has trouble with is with folders that were removed on the server - they are still displayed in the local tree. I had a few instances that only throw an error when I click the "orphaned" folder. Not sure what is going on here.

[jaysmithgit]

Hi Axel,

Thanks for the further new versions. As previously mentioned, I can't test until tomorrow, though last night I did get the test bed set up. Your own testing sounds promising. I am looking forward to testing it. I will come back to you as soon as I can.

Jay

[jaysmithgit]

TESTING of "pre38" for "FORCE TREE REFRESH" function

SUMMARY:

1) The function initially works (to open the folder structure), but only for a few upper levels. Then it silently dies. 95% (or more) of the folders-that-contain-folders are never opened.

2) As part of its function, Force Tree Refresh is actually causing the reading of the mail messages (to get the count), as if a user manually clicked on a folder-that-contains-mail, which results in displaying the message count for folders-that-contain-mail. The counting is UNNECESSARY (for me) and could be part of the reason that the function dies. I do not need the messages to be counted -- I only need the folders to be opened to expose the levels of folders below (so that QF "go-to, move, and copy" functions will be able to "see" the possible folders). I do not know how many total messages our IMAP server is holding, but it is certainly over 100,000 (more than 20 years of mail), and it is not necessary to count all of them (even if the "Force Tree Refresh" did not die before getting to the bottom levels of folders).

=-=-=-=

This testing was done with a new/fresh TB 78 installation, against our actual IMAP server folder tree.

I am using the latest version you supplied: QuickFolders-wx-5.6pre38.zip

Under "Subscribe", I have (as always) turned OFF "only show subscribed". The Subscribe dialog does correctly show the entire structure. Thunderbird IS seeing the whole IMAP structure in the Subscribe dialog. (But that does not seem to help QF see the IMAP structure.)

=-=-=

Force Tree Refresh starts, but appears to stop after one or two minutes. During that couple of minutes, 99% of the activity seems to be taken up by counting messages in folders-that-contain-mail. Even after 20 minutes, there is no indication of any further activity.

• There are no error messages that I can see.

• The Activity Manager does NOT show any activity related to this function. (It does show activity such as when I am writing a draft of this posting, using Thunderbird's Compose, and the Compose is auto-saved to Drafts, so Activity Manager IS basically functioning.)

• I did turn on QF Debug Mode, but I am not sure what that is supposed to do. Debug mode did not do anything or show anything that I could see.

• While the "Force Tree Refresh" is operating...

  • I did see activity in Thunderbird's, bottom status bar.

  • I did see folders being visibly created in the left-side-folder-pane.

  • I did see mail messages being counted <<< THIS IS NOT NECESSARY AND IT REQUIRES THAT THE CONTENTS OF EVERY FOLDER IS ATTEMPTING TO BE READ. If it worked to open all the levels of folders, it would attempt to read probably at least 100,000++ emails to count them. That is UNNECESSARY.

NOTE: AFTER the test, In the left-side-folder-pane-tree, when a user MANUALLY clicks on a folder-that-contains-folders, the folder opens and Thunderbird reads [the IMAP?] and displays the folders inside the folder that was just clicked (just as normal). TB does NOT count mail messages at that time (that is normal -- TB only normally counts mail messages when the user manually clicks on a folder-that-contains-mail. I am only trying to replicate the function of opening the folders, but without counting messages.

• "Force Tree Refresh" opens a few levels of folders, but ONLY a few. There exists in the IMAP several more levels below the level at which the function dies.

• It appears that "Force Tree Refresh" is dying before it gets very deep. Perhaps it is running out of allocated task/tree memory or the iterator simply can't handle the depth, or ???. (The tree is both broad and deep.) I do not see any error messages.

The following is a brief summary of what the final result looks like. If you would like me to email directly to you screen shots of what it really looks like, I will be happy to do that.

user@foo.com
   Inbox       ALREADY VISIBLE, UPDATED COUNT
   Sent        ALREADY VISIBLE, UPDATED COUNT
   (etc.)

   public  ALREADY VISIBLE (CLOSED), OPENS IT & CREATES THE FOLLOWING
      folder-that-contains-mail-1      CREATES & COUNTS
      folder-that-contains-mail-2      CREATES & COUNTS
      folder that-contains-folders-X   CREATES & OPENS
      folder that-contains-folders-Y   CREATES & OPENS
         folder-that-contains-mail-a      CREATES & COUNTS
         folder-that-contains-mail-b      CREATES & COUNTS
         folder that-contains-folders-x   CREATES, does **NOT** OPEN
         folder that-contains-folders-y   CREATES, does **NOT** OPEN
             NOTHING BELOW/INSIDE THIS IS CREATED, OPENED, OR DISPLAYED
             Several levels exist below this point.

If I manually click on one of those last "folder that-contains-folders-y" level of folders, it WILL open and display the folders inside it, as usual when manually clicking. Thus they do exist and Thunderbird can see them on the IMAP when I click them with the mouse.

However, the "Force Tree Refresh" does not open or display the folders inside that level or any farther down.

I have repeatedly tried running "Force Tree Refresh" and it does not appear to do anything further, though there is some extremely fast text displayed in the bottom status bar (much too fast to read).

Restarting Thunderbird and trying again to do "Force Tree Refresh" does not result in any further folders (displayed in the tree) created or counted.

Running Thunderbird from the command line (just simply "thunderbird" in our installation) typically results in displaying (at the command line) a few error and debug messages (nothing important). However, NO additional error/debug messages are displayed as a result of running "Force Tree Refresh".

=-=-=

I hope that this information is useful. If you have specific questions, I will do my best to answer them.

If you wish me to run a different test, I will be happy to do so.

Jay

[RealRaven2000]

Interesting test results - I would have guess that running it repeatedly it would "go deeper" - but maybe the tree nodes have to be actually expanded for this. I was hoping that this would not be necessary. You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise. I think the status bar is updated every time another folder is expanded. I don't think Thunderbird Activity Manager is aware (or cares) that I am the user is opening folders, and it doesn't necessary build the index file unless it is the current folder. I need ti rethink, maybe there is a different thing I can do.

[RealRaven2000]

Force Tree Refresh starts, but appears to stop after one or two minutes. During that couple of minutes, 99% of the activity seems to be taken up by counting messages in folders-that-contain-mail. Even after 20 minutes, there is no indication of any further activity.

the idea is that it adds missing nodes while iterating the folders it already knows. But if you do it repeatedly it would iterate the new folders, too and I was hoping that it would then drill deeper. you can watch what it does if you enable "debug mode" in QF advanced and keeping the JS error console open.

[jaysmithgit]

A couple updates...

1) I experimented further and manually opened a folder (just one level) that contains many more folders (some containing messages and some containing further-deep folders). I then again did "Force Tree Refresh". When I manually opened the folder, that triggered Thunderbird to read the IMAP and make its contents visible in the tree. THUS the effect in this state of running Force Tree Refresh was only to count the messages.

2) I was very wrong about the number of messages on our system. The first level of the folder (that is the only level that opened; and I was the one who opened it, not QF) just mentioned above, contained folders-containing-mail that had over 140,000 messages in it. I think the counting just done took about 90 minutes (I was not watching closely). Thus our total message count for more than 20 years is probably closer to 250,000 messages. That's a lot to count.

[jaysmithgit]

Interesting test results - I would have guess that running it repeatedly it would "go deeper" - but maybe the tree nodes have to be actually expanded for this. I was hoping that this would not be necessary. You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise. I think the status bar is updated every time another folder is expanded. I don't think Thunderbird Activity Manager is aware (or cares) that I am the user is opening folders, and it doesn't necessary build the index file unless it is the current folder. I need ti rethink, maybe there is a different thing I can do.

!!! YES, YES, YES: "...but maybe the tree nodes have to be actually expanded for this." That is what I am trying to do this whole time. Expand ALL TREE NODES and then TB should "see" them and make them visible to QF for move-to/copy-to/go-to.

Counting messages is completely optional for me (and would take a long time).

[jaysmithgit]

the idea is that it adds missing nodes while iterating the folders it already knows. But if you do it repeatedly it would iterate the new folders, too and I was hoping that it would then drill deeper.

Unfortunately, it is not drilling deeper. TB seems not to recognize that it is being asked to open the next level of nodes.

What is needed is a recursive command: "If folder is a folder-that-contains-folders, open it...".

[jaysmithgit]

You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise.

Actually, the function did not close any folders. At this time, it leaves open any folders that it opened (or which were already open). I am sorry that I failed to say this in my original test results report.

I think the function dies before it gets to the stage of closing folders.

Actually, closing the folders is not very important to me (but perhaps it would be to other users). Our hierarchical structure is such that I can close/hide thousands of folders by closing only a very few folders at a higher level.

[jaysmithgit]

Thank you for telling me about the JS error console. I had not used it in years and had forgotten about it.

That showed a couple interesting things:

When running "Force Tree Refresh"...

MESSAGE #1 in the JS error console:

QuickFolders 18:39:47.379 [141301 ms]
Parent panel {QuickFolders-PreviewToolbarPanel} is not on screen; moving current folder button for tabMode: folder [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/L10nRegistry.jsm :: L10nRegistry.loadSync :: line 658" data: no] L10nRegistry.jsm:658:19

MESSAGE #2 in the JS error console:

QuickFolders 18:39:59.67 [11688 ms]
switched TreeView mode= all

ALL THE REST OF THE MESSAGES (HUNDREDS)... in the JS error console: All the following (this is just a small sample) are terminal folders containing mail NONE of the following are folders that contain folders !!!

First the user's own area:

QuickFolders 18:39:59.94 [27 ms]
theTreeView.selectFolder(Inbox) => index = 1, hasSubFolders = false, open = false QuickFolders 18:39:59.118 [24 ms]
theTreeView.selectFolder(Trash) => index = 5, hasSubFolders = false, open = false QuickFolders 18:39:59.141 [23 ms]
........ etc.

Then the "public/" IMAP folders:

theTreeView.selectFolder(jay_orders) => index = 21, hasSubFolders = false, open = false QuickFolders 18:39:59.434 [17 ms]
theTreeView.selectFolder(_spam_that_shouldnt_be) => index = 12, hasSubFolders = false, open = false QuickFolders 18:39:59.448 [14 ms]
theTreeView.selectFolder(stock) => index = 25, hasSubFolders = false, open = false QuickFolders 18:39:59.471 [23 ms]
theTreeView.selectFolder(jay) => index = 19, hasSubFolders = false, open = false QuickFolders 18:39:59.489 [18 ms]
theTreeView.selectFolder(auctions) => index = 15, hasSubFolders = false, open = false ........ etc.

Again, ALL the folder names in this listing (of hundreds) are only the "terminal" (final in the structure) folders that actually contain mail.

There are NO folders that contain folders in this list.

Either the function is simply not logging the "opening" of folders, or it is not trying to open them???

[RealRaven2000]

Just to document what I am doing - the folder iterator has to start from somewhere - so it iterates all folders that are seen and tries to open them in the tree. The "open=false" log actually shows whether the folder is expanded in the tree. The code I am using to attempt a "drill down" is taken from Thunderbird's _toggleRow function

https://searchfox.org/comm-esr78/source/mail/base/content/folderPane.js#1284

In my tests I was able to search for (empty, so not mail containing) folders I added on Imap straight away even when the parent folder was collapsed. I don't really know what else to do, guess I need to find a Thunderbird Core developer who is interested in this topic

[RealRaven2000]

Interesting test results - I would have guess that running it repeatedly it would "go deeper" - but maybe the tree nodes have to be actually expanded for this. I was hoping that this would not be necessary. You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise. I think the status bar is updated every time another folder is expanded. I don't think Thunderbird Activity Manager is aware (or cares) that I am the user is opening folders, and it doesn't necessary build the index file unless it is the current folder. I need ti rethink, maybe there is a different thing I can do.

!!! YES, YES, YES: "...but maybe the tree nodes have to be actually expanded for this." That is what I am trying to do this whole time. Expand ALL TREE NODES and then TB should "see" them and make them visible to QF for move-to/copy-to/go-to.

That's exactly what the code tries to do. It should at least expand all the folders that are logged with "open=false" and thus expose their direct children. Anything that is listed in your log file like this should have a complete list of children afterwards. but I guess maybe I need to force to "download messages" on the folder (even if it is empty apart from subfolders). I have a download all messages command on context menu for the Imap folders, but at least on gmail it always tries to update my Inbox and Sent folders, which takes ages.

[RealRaven2000]

the idea is that it adds missing nodes while iterating the folders it already knows. But if you do it repeatedly it would iterate the new folders, too and I was hoping that it would then drill deeper.

Unfortunately, it is not drilling deeper. TB seems not to recognize that it is being asked to open the next level of nodes.

What is needed is a recursive command: "If folder is a folder-that-contains-folders, open it...".

it won't work that way around. because you do not know the fires part "is a folder-that-contains-folders" without opening it first. So my algorithm actually opens everything that has "open=false" in the log in order to force expanding it from the remote server - at least that's my theory. The actual communication with the Imap server is hidden to me as it happens super low level - one would need a C++ compiler to see what happens there and I just haven't got the time to set up a complete Thunderbird build environment.

There are NO folders that contain folders in this list. Either the function is simply not logging the "opening" of folders, or it is not trying to open them???

It tries to open all folders, regardless of whether they are expanded or not. It just works with the data that Thunderbird volunteers to give it.

[jaysmithgit]

It tries to open all folders, regardless of whether they are expanded or not. It just works with the data that Thunderbird volunteers to give it.

This is very puzzling because on the GUI, I can visually see the first-level-at-that-point-in-the-process of folders-that-contain-folders that the function is not "seeing", thus which the function is not trying to open.

Is it possible that some how the function's code is dying because of some environmental factor, such as limit to the number of threads, processes, forks, memory, etc. available to that function. As the tree spreads out, the management of the iteration becomes much, much more complex. The iterator has to "protect" itself and thus can't just open everything in sight, and then open everything below that, and everything below that, because in a very large structure either the iterator, or the subsequent "count messages" processes would run out of resources.

Of course, I don't have the knowledge to know, but my thinking is that the reason that the iterator is not "seeing" the deeper folders-that-contain-folders is that the iterator has already died by that time. Perhaps not a problem that Thunderbird is not making the information available, but that the iterator can't see it because it is already dead. ???

Is it possible to instruct the iterator, or to use an iterator designed for deeper/broader structures, to follow only a few branches of the tree at a time, and at each subsequent branch, have another such limit on the number of branches? Then when it hits the terminal end of the branches, it rises back up to the previous level and looks for more unopened branches?

[RealRaven2000]

It tries to open all folders, regardless of whether they are expanded or not. It just works with the data that Thunderbird volunteers to give it.

This is very puzzling because on the GUI, I can visually see the first-level-at-that-point-in-the-process of folders-that-contain-folders that the function is not "seeing", thus which the function is not trying to open.

it won't "die" without an exception (which my code catches and reports) or Thunderbird hanging. New version:

QuickFolders-Lic-5.6pre93.zip

I added an endpoint to make sure. at which point it should say something like

QuickFolders 15:47:41.316  [26 ms]   
 refreshTree() iterated all accessible (62) folders.

Run it and then save the complete log output in a text file (that way you can search for the folder's name you refer to) using the "export visible messages to..." command.

[RealRaven2000]

P.S. feel free to upload the log file as an attachment here if you are comfortable with it.

[jaysmithgit]

It should at least expand all the folders that are logged with "open=false" and thus expose their direct children. Anything that is listed in your log file like this should have a complete list of children afterwards. but I guess maybe I need to force to "download messages" on the folder (even if it is empty apart from subfolders).

Just to be sure that there is no miscommunication in my test results report, the JS error console messages (of which I gave an example) contain absolutely NO listings for folders-that-contain-folders.

Thus, when you say "It should at least expand all the folders that are logged with "open=false" and thus expose their direct children." ... I want to be sure you are aware that it is NOT logging ANY folders-that-contain-folders.

Thus, if it is not logging them, perhaps there is some aspect of the function that is not behaving as expected.

I don't really know what else to do, guess I need to find a Thunderbird Core developer who is interested in this topic.

There are a couple things that support the possibility that there is something wrong (in regard to folder depth) either with the Thunderbird code that communicates with the IMAP, or something wrong with Thunderbird's [ _toggleRow] (https://searchfox.org/comm-esr78/source/mail/base/content/folderPane.js#1233) function (that you mentioned above).

1) In at least 10 years, "nobody" has had an answer to this -- though you have certainly done more work on it than anybody has ever tried to do.

2) On OLD Thunderbird (long before 78) the "Compact All Folders" function dies (silently, though I have not ever tried to watch in the JS error console). I have not wanted to test it on TB 78 as it can take a few hours before it dies and it puts a very heavy load on the server.

I hope it is possible to interest a Thunderbird Core developer in this topic.

[RealRaven2000]

Did you run the new version? A complete log would be helpful. "folders that contain folders" would mean the function doesn't even iterate any second level folders at all? so you basically only see folers at the root and nothing any deeper? e.g. f1 / f2

if you see f2 then you see f1 (which is a folder that contains f2). Not quite sure what a "folder that contains folder" means. Probably a folder without emails... But in my test these show too. So a complete log file and the path of one of these folders that is not seen would bring some clarity.

[jaysmithgit]

Did you run the new version?

I have not yet run QuickFolders-Lic-5.6pre93.zip. It may be a day before I can run it.

A complete log would be helpful. "folders that contain folders" would mean the function doesn't even iterate any second level folders at all? so you basically only see folers at the root and nothing any deeper? e.g. f1 / f2

I will be happy to supply the complete log, but I will either have to email it to you directly (I have your address already from our previous conversation) or I can post it in my webspace where you can download it. I do not want to post it publicly here.

if you see f2 then you see f1 (which is a folder that contains f2).

I do see f1, f2, f3, f4 ... but I see NO more below f4 (unless I manually click to open f3, which then properly reads f4 and then manually click to open f4, which then reads f5, etc, etc.).

Not quite sure what a "folder that contains folder" means. Probably a folder without emails... But in my test these show too. So a complete log file and the path of one of these folders that is not seen would bring some clarity.

Unfortunately, we are continuing to have the communication problem, that we had from the very beginning, with the meaning of "folders". This communication problem may be at the core of why it is difficult to make progress?

There are TWO types of folders, or you could say that folders can have TWO statuses.

1) Folders that only contain mail (results in .msf files) 2) Folders that contain other folders (results in .sbd directories)

Using your (f1, f2, etc.) illustration I will try again to describe:

In this example below, if I use the "Force Tree Refresh" function, the f4 level is visible, but f4 folders have NOT been opened, thus all the f5 folders (and below) are NOT visibile.

f1       (user@foo.com)
    f2a  Inbox    (that users folders)
    f2b  Sent      "   "

    f2c "public/"   (the shared IMAP structure)

        f3a     [only contains folders]
              f4a    [only contains folders]
                  f5a   (folder that contains mail)   NOT VISIBLE
                  f5b   (folder that contains mail)   NOT VISIBLE
                  f5c   (folder that contains mail)   NOT VISIBLE
              f4b    [only contains folders]
                  f5m  (folder that contains mail)   NOT VISIBLE
                  f5n   (folder that contains mail)   NOT VISIBLE
                  f5n   (folder that contains mail)   NOT VISIBLE
        f3b     [only contains folders]
              f4c    [only contains folders]
                  f5x   (folder that contains mail)   NOT VISIBLE
                  f5y   (folder that contains mail)   NOT VISIBLE
                  f5z   (folder that contains mail)   NOT VISIBLE
        f3c     [only contains folders]
              f4d    [only contains folders]
                   f5r   [only contains folders]   NOT VISIBLE
                        f6a      [only contains folders]   NOT VISIBLE
                               f7a      [only contains folders]    NOT VISIBLE
                                    f8a     [only contains folders]   NOT VISIBLE
                                         f9a   (folder that contains mail)   NOT VISIBLE
                                         f9b   (folder that contains mail)   NOT VISIBLE
                                         f9c   (folder that contains mail)   NOT VISIBLE

[RealRaven2000]

Good example. But f3a and f4d also "only contain folders" and they are displayed, so what is different to f4a / f4b? The depth of level maybe?

[jaysmithgit]

Good example. But f3a and f4d also "only contain folders" and they are displayed, so what is different to f4a / f4b? The depth of level maybe?

In your question, all of f3a, f4d and f4a, f4b ARE visible. There is no difference between the type of folder they are; but f3a is a level above f4a, fb4 folders [f3a CONTAINS f4a, f4b]. (I only wrote "NOT VISIBLE" for the items that were not visible. All the others ARE visible).

In that illustration,the digits (3,4) represent the levl from the top. The letters (a,b,c) are just identifiers to be able to talk about, in this conversation, a particular line in the structure.

All the f3 folders are at a certain level, i.e. one level deeper than f2 . All the f4 folders are one deeper lever than f3 folders. (All the f3 folders are contained in one or more f2 folders; all the f4 folders are contained in one or more f3 folders.)

In that illustration, all the f3 and f4 folders are by coincidence folders-that-only-contain-folders. It is just representation of one sample hierarchical structure. Most private users probably do not have a structure that is so deep, but large companies probably have an even-deeper structure; we are only a small company.

You could add items f3d and/or f4e that are folders-that-contains-mail, last two lines in the illustration below:

        f3c     [only contains folders]
              f4d    [only contains folders]
                   f5r   [only contains folders]   NOT VISIBLE
              f4e    (folder that contains mail)
        f3d    (folder that contains mail)

Unfortunately, I don't know anything about other possible methods of installation, but in our installation, we CANNOT put mail directly into a "folder-that-contains-folders". In our installation it is only possible to put mail into a "terminal", mail-containing, folder-that-contains-mail, that is the final folder at the "end of the path".

=-=-=

Perhaps If we used the terminology of computing that I was taught, then we might not have so many complications in discussion.

However, I suspect that the communication problem is that there are types of Thunderbird installations which do allow "folders-that-contain-folders" that are ALSO ALLOWED to DIRECTLY CONTAIN MAIL. (If that is true, that is unfortunately very complicated.)

"Folders-that-contain-mail" are "files" in computing (and on the IMAP server) in our installation. These result in ONLY .msf files being created.

"Folders-that-contain-folders" are "directories" in computing (and on the IMAP server) in our installation. These result in BOTH .msf files AND .sbd directories being created.

[jaysmithgit]

it won't "die" without an exception (which my code catches and reports) or Thunderbird hanging. New version: QuickFolders-Lic-5.6pre93.zip I added an endpoint to make sure. at which point it should say something like

QuickFolders 15:47:41.316  [26 ms]     
 refreshTree() iterated all accessible (62) folders.

Run it and then save the complete log output in a text file (that way you can search for the folder's name you refer to) using the "export visible messages to..." command.

I have run the latest (for this work) test version (93). I am sorry that I was not able to do this for a few days.

For the first run of the function, I started the test with only USER@mail.FOO.com opened, showing on screen ONLY 1) The user's Inbox, Sent, Trash, etc. and 2) the IMAP public/ folder (closed).

The function did open/expose 1) the IMAP public/ folder and 2) some/most (?) of the first level of folders under that and 3) only some of the next level. The difference of what it did and did not attempt to open/expose seems to be whether or not the lower levels had previously been opened/exposed -- if they had not been, then they were not this time either. (see notes farther below).

There was no significant improvement in the results. Thinking that it might iterate further down into the structure if I ran the function multiple times, I ran the function 5 times and saved the new log after each time so that you can compare them. I really don't think there is any difference (other than the first log has more start-up preferences-recording information). I ran kompare on pass 1 vs pass 5. Though it hard to tell because the timestamp makes the lines different, but I don't think the two files are actually different in terms of what the function is supposed to be doing.

I do not think this version attempted to count the number of mail messages in "folders that contain mail". However, it is hard to know for sure, because the function only acted on "folders that contain mail" when those folders had already previously been opened/exposed (and thus already counted) by either me manually opening/exposing them or a previous version of your add-on opening/exposing AND counting them.

I could not detect that this test of the function opened/exposed and counted, any previously UNopened/exposed folders.

There is one important point I need to make that might (?) be helpful: The only time that the function is actually attempting to open, below the first one or two levels, any "folder that contains folders" is when the contents of the "folder that contains folders" had been previously manually opened/exposed by me clicking around on stuff on the GUI.

(IF in the user's .thunderbird IMAP structure [i.e. .msf files and .sbd directories], I delete the structure [i.e. all the .msf and .sbd files] that has so far been created by opening/exposing, then we will go back to "new, unopened" state, and most of what the function is currently able to see and thus try to open, will not been seen/opened by the function. The structure I am talking about is in: /XXX/home/USERNAME/.thunderbird/12345678ABC.default-release/ImapMail/mail.FOO.com/public.sbd/....

In a moment I will email you the full log files. I am not comfortable posting that structure information publicly.

I will also attempt to separately email you the file structure listing (ls -R) on the IMAP server so you can get some sense of what the function is not seeing (99% of the structure). If you have a better suggestion of how to create a visual tree listing of what is on the IMAP server, please instruct me.

Thank you for your continued efforts on this. I really appreciate it and if the function can be made to work, I think it will make your add-on of tremendously greater use to installations with large IMAP structures.

[jaysmithgit]

Axel,

I wanted to say that as I have become more accustomed to using QuickFolders Pro, I am really impressed with how it has improved my productivity, even though I was previously using the "other" add-on. (Other than solving the "make IMAP folder structure visible" problem, the only thing that would make it even better would be "keyboard access to the Recent List".) QuickFoldlers has become natural to use -- much more so for me than that "other" add-on (Nost...). So, I want to say a very loud... Thank You! I know that the high level at which you have developed and maintained QuickFolders takes a huge amount of work and dedication.

I sure hope you will be able to come up with a way to solve the "make IMAP folder structure visible" problem. Please let me know if you need any further information or examples or testing from me.

Jay

[fredstluka]

Jay,

3 things, including hopefully a workaround for your problem:

1. I've really enjoyed reading this conversation between you and     Axel.  You've both been very patient and explanatory about all     the details.  It's great to be using a product with such dedicated     and mature devs and users.

2. I LOVE hearing that you use Thunderbird at such a huge scale.     I thought I was a big user since I have every email that I've sent     or received for the past 20-30 years.  But I only have 611 folders     containing about 500,000 messages for a total of 7 GB of storage

3. Do you ever search the bodies of all of your messages?  I use POP,     not IMAP, so this may not work for you, but...

       I had a problem for a year or more, a while back, that my .msf files     would suddenly start getting deleted, about one per second.  I     always had to exit Thunderbird to stop the deletions, and then     move through all the folders (as you described) to recreate all the     .msf files.  I got to be pretty quick at it by hitting alternating     right-arrow key (to open a parent folder if I was on on) and     down-arrow key (to move to the next folder), but it was still a real     chore to do so.

       I discovered after a while that doing a search across my entire     tree for messages containing a random string like "crapola", even     if I found no messages, would recreate all of the .msf files. Give     it a try, and let me know if it works for you.  I think it may have     worked on both Mac and Windows.  On Mac, I used:     - Thunderbird | Edit | Find | Search Messages...  (Cmd-Shift-F)

       It might only work if you search the bodies, not the subjects,     To, From, etc.  I don't remember details, since I haven't had to     do it in a while.  You'll see status messages at the bottom of the     search window naming the folders that it's re-indexing, so you'll     have a reasonable progress indicator.

       If this doesn't work for you, perhaps there some other global     operation you could try, like deleting the global message index     and then doing a search, so it has to recreate the index.  I     haven't had to do that it years, but back in the days before the     index was a SQLite file, I used to delete the old index file with     no ill effects.  You should be able to do the same with the new     global-messages-db.sqlite.

       I just did a quick Google search and found:     - https://support.mozilla.org/en-US/kb/rebuilding-global-database     where the officially suggest deleting it to solve a number of     different problems.

       BTW, since you're dealing with .msf files, you may like some of     the Mac/Unix/Linux scripts I wrote during that time:     - http://bristle.com/Tips/Mac/Unix/tb_msf_count     - http://bristle.com/Tips/Mac/Unix/tb_msf_count_loop     - http://bristle.com/Tips/Mac/Unix/tb_msf_count_beep     and maybe these too:     - http://bristle.com/Tips/Mac/Unix/tb     - http://bristle.com/Tips/Mac/Unix/tbfilters     - http://bristle.com/Tips/Mac/Unix/tbw

Good luck! --Fred

Fred Stluka -- http://bristle.com -- Glad to be of service! Open Source: Without walls and fences, we need no Windows or Gates.

On 5/18/21 6:14 PM, Jay Smith wrote:

Axel,

I wanted to say that as I have become more accustomed to using QuickFolders Pro, I am really impressed with how it has improved my productivity, even though I was previously using the "other" add-on. (Other than solving the "make IMAP folder structure visible" problem, the only thing that would make it even better would be "keyboard access to the Recent List".) QuickFoldlers has become natural to use -- much more so for me than that "other" add-on (Nost...). So, I want to say a very loud... Thank You! I know that the high level at which you have developed and maintained QuickFolders takes a huge amount of work and dedication.

I sure hope you will be able to come up with a way to solve the "make IMAP folder structure visible" problem. Please let me know if you need any further information or examples or testing from me.

Jay

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RealRaven2000/QuickFolders/issues/154#issuecomment-843600836, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADKWRJS3AVAZ76LIBE7AO3TOLRGDANCNFSM44DNHQ5A.

[fredstluka]

Jay,

Have you had a chance to try my idea about doing a global search and/or deleting the global search index so it rebuilds itself?  Any luck?

--Fred

Fred Stluka -- http://bristle.com -- Glad to be of service! Open Source: Without walls and fences, we need no Windows or Gates.

[jaysmithgit]

Jay, Have you had a chance to try my idea about doing a global search and/or deleting the global search index so it rebuilds itself? Any luck?

Fred, Thank you for the suggestion. At first I did not understand it because I was asking myself "what global search?". Then I forgot about it.

Now, with your reminder, I understand (thanks Google). In all these years, I had no idea there was a global search function -- it is not present in my GUI!! However, in all these years, the first thing we do when installing a Thunderbird instance is to turn OFF Global Search and Indexer in the preferences -- thus I had never seen the search box in the GUI! :-)

Back in the day (at least 15 years ago), our procedure to turn off the indexer upon installation was because of the high resource demands on the server. There was a time when it really was a pig. Multiple users, each doing all that indexing against many thousands of email folders and over 100,000++ email messages, takes a lot of resources on a server that is also serving several large vmware images, a big database, and everything else going on. It's a big, fast, "real" server with fast disks (but RAID-5 which may be slower) and 64 GB of RAM, 16 or 32 (I forget which) processor cores, etc., etc., but it still bogs down sometimes as it is. I suspect when we replace the server next time around (it is about time now!), we may be able to get enough speed gains from the new tech that it will make more sense to leave on the TB indexer by default.

However, I think your idea is definitely worth testing. That may explain why our Thunderbird instances (users/seats) don't automatically self-discover the entire folder tree.

Soon I have to install a new seat, so I will try leaving the Global Search and Indexer on and see what happens. I will have to do it on a weekend night (we run 7 days a week) because if it tries to index 100,000++ emails it will take a while and could bog down the server pretty badly depending upon how the Indexer is written. Do you know how "nice" it is -- does it (TB's Indexer) running on a client workstation, have a way to detect server load or response time and back off if the server is too heavily loaded / too slow?

If this solves the problem of discovering the entire folder tree, we could start out by leaving on the Indexer when installing a new Thunderbird, then when it is "done" indexing (for the time being), turn it off. It could be turned on again every couple of weeks to freshen things up. I am not sure how well it would react to that, but it might be worth a try.

Thanks again for the idea.

[fredstluka]

Jay,

Good luck with it!  For me, Thunderbird is a little sluggish doing a global search of bodies, so I don't know how good a job it does or running on a low priority thread.

I use it VERY often to do global searches of "From, To, CC, and BCC" whenever I want to quickly see all email correspondence I've had with a specific person for the past 20-30 years.  And it runs fast in that case.  I'm on a MacBook Pro laptop, and it took about 10 seconds to find all 12,486 emails to/from my brother Tom from all of the 500,000 or so emails stored in 611 folders and consuming 7 GB of storage.

Keep in mind though that such a search (of headers, not bodies) is searching only my 611 MSF files.  Not searching the entire message files and creating the MSF files, which is what you'll want it to do. I kicked off a search of bodies just now, as I'm typing this message. It took about 2 minutes, and DID make Thunderbird sluggish.  I could see the difference even in things like moving around the text I was typing with arrow keys, Backspace, etc.  Noticeably slow.

Better to do it over a weekend or some other low-demand time.

--Fred

Fred Stluka -- http://bristle.com -- Glad to be of service! Open Source: Without walls and fences, we need no Windows or Gates.

[RealRaven2000]

I asked John for some code he wrote for a GLODA patch that iterates folders, here is a link to the code to look at later:

https://searchfox.org/comm-central/source/mail/components/extensions/parent/ext-messages.js#744-769

[jaysmithgit]

UPDATE from the OP...

To some extent, this problem may not be the problem I thought it was. But there may be a different problem at large scale.

(I am still on TB 78.7.1 and QF 5.5.2 PRO LICENSE -- I have not attempted upgrading either in the last couple of months as I knew I did not have the time to deal with potential problems if something did not work. And I am working in a production environment where TB and QF are extremely important.)

(Background: Each of our users has the same access to an IMAP structure containing tens of thousands of folders, arranged in a hierarchical structure 6, 7, 8 levels deep. While most folders either contain only modest quantities of messages OR contain only deeper folders, there are some individual folders that contain many tens of thousands of messages; the largest I saw had something over 85,000 messages.)

One of the underlying problems/issues/reasons for this original thread was to solve the problem that I thought that TB, by itself, had no way to expose the complete folder tree, to thus register all the folders in TB's database, so that QF could see and thus file into any folder in the tree. This was relevant in new installations (or just new profiles because of corruption OR because the panacea.dat file has corrupted, or results in major slowing of the system, and panacea.dat needs to be rebuilt) of TB. The only workaround that I knew about (using "ancient TB") was to manually 'walk' the entire folder tree with the mouse. I did not / do not care about message counts, etc., I just needed to expose / register the folders into the TB database. Still, just manually exposing / registering the entire folder tree (without getting any message counts) took about 5 hours per user. This had to be done for every new user. Also, because of somewhat regular "ancient TB" panacea.dat corruptions, it had to be additionally done for every user approximately once every 18 months -- heavy users, every six months or so. (And, when one user added a folder structure [folder containing folder containing folder, etc.] the the existing structure, this was only exposed / registered for the other users if they other users also re-walked the folder tree (hours of work).

However, my assumption that TB had no way to expose the complete folder tree and register it in the basis was incorrectly based on my experience with ancient TB -- specifically that the "Compact (All) Folders" menu command was (back then!!) not robust enough to successfully complete the process on a folder structure as broad and deep as mine (and with as many emails). I don't know if the past failures were a memory management problem or a thread management problem, but the compacting process used to die after an hour or two -- and locked up TB requiring that TB be 'killed'.

UPDATE... A couple days ago, due to a "keyboard mishap", in TB instead of typing ALT f v (Print Preview), I typed ALT f f. That is, unfortunately, "Compact (All) Folders". It appears that there is no graceful way to stop that process, so I just let it run. The "Compact (All) Folders" command appeared to successfully complete after running for 4-5 hours and processing what I estimate to be nearly a million (1,000,000) emails. THE GOOD NEWS IS THAT THIS PROCESS APPEARS TO HAVE RESULTED IN EXPOSING THE ENTIRE FOLDER STRUCTURE AND REGISTERING ALL THE FOLDERS IN TB's DATABASE IN SUCH A WAY THAT QF CAN SEE THEM FOR GOING TO FOLDERS AND FOR MOVING EMAILS.

Note that the "Compact (All) Folders" command, done by each user, will also, for that user, expose / register newly created folder structures made my other users. Such commands can be done, every few days or weeks, as one leaves the office at the end of the day and allowed to run overnight.

For me, the fact that the "Compact "All" Folders" actually worked, and seemed to complete successfully, was a total and unexpected surprise. I now realize that I should have re-tested that feature, and not made assumptions, since I had previously been on "very ancient" TB and am now on 78.

At the same time, it is very interesting to me that the TB Core folks with which Axel had discussed the exposing / registering need/problem apparently never suggest to Axel "just have the user do a "Compact (All) Folders"!! (Whether or not the Compact function should take 4-5 hours, and whether it should be more controllable, is a completely different question for another time and place.)

However, the BAD NEWS seems to be that, since the compacting / exposing / registering, in the last couple of days, instead of being instantaneous when doing a GO-TO or MOVE command QF takes between 1 and 5 seconds to respond and present the requested list of possible targets. Most of the time it is in the 2-3 second range; it is now seldom less than 1 second (which would be okay if consistent), but sometimes is up to 5 seconds:

Repeatable process:

(I have the PRO LICENSE) Type (for me, I am not using the default keystrokes) "s" (save/file-into) or "g" (go to), immediately followed by the target mail folder string.

If I ONLY type (for me) the "s" or "g" then response is immediate. But if I do start typing further characters, then I have the delay of 1-5 seconds as noted above. From that point on, it behaves normally / quickly. I can arrow down into the results list just as quickly as before.

I think what has happened is that by exposing/registering many thousands more (I am not sure how many more that I had not already exposed/registered manually using the GUI) folders, QF now has to parse a much larger list.

When this delay first occurred, the delay was so significant that I thought TB had locked up and I started trying type other things and use the mouse. However, then TB caught up to my typing and mouse use -- causing a mess resulting in a various problems because my focus was then not where I thought it was. (My fault.)

In the couple days since then, the delay seems to be getting less -- maybe my imagination. Perhaps there is some "remembering" or AI aspect at play which helps TB support QF's query of the TB database.

If I experience further improvement -- or worse performance -- I will report further. However, please be aware of the "long" amount of time it may take the Move and Jump commands to function with a very large, fully exposed / registered folder structure.

Jay

[RealRaven2000]

OK, good findings - I highly encourage you to try 5.7 on one of your installations (you can always roll back as nothing changes in the configuration that will impact rolling back to 5.6). Maybe there is a way we can optimize the searching so that you can keep typing, it just needs to be done in an asynchronous manner (I believe there may be another issue that mentions this).

check out #179 for this one (shorter comments there, if you can)

[fanis]

I came to this thread after looking for a solution to a similar problem I've been having with TB 102 and frequently corrupted folders, which lead me to have to repair them one by one. I do that by browsing to the folder in the folder tree listing, which makes TB do the repair if the folder is corrupt (I don't know if it's corrupt in advance and only find out if a filter I have on new mail fails to move mail to the folder in question and instead leaves it in the Inbox).

While searching for a way to programmatically visit every folder to trigger the repair, I noticed the promising debug function Force Tree Refresh but it doesn't seem to be working for me in Local Folders where I have all my (hundreds) subfolders.

@RealRaven2000 , I understand this thread is from 1 year ago but I can't help but ask if there was any further thought or development around this particular issue and what seems like a hacky fix, namely visiting all subfolders in sequence for the repair to work.

I realize this is curing a symptom rather than the underlying issue, which I have yet to discover, but it will help as a patch when the issue arises in the mean time.

Obviously I understand this is not an issue with QuickFolders itself, which I confirmed by disabling the extension and still having the issue appear in TB 102, so I don't expect much motivation in working on anything related but instead only hope for it.

[jaysmithgit]

Hi fanis,

I am the OP of this thread.

I am still behind the curve on TB versions (I am still on TB 78.7.1) so what I have to say is certainly not directly helpful to you. [TB is mission critical for us, so when something is working well enough, I am very hesitant to upgrade.]

However, I have not been told by anybody that newer TB yet solves this problem that I originally described and that I believe you are encountering (needing a way to systematically open/expose/register/read all folders). I don't know anything about corrupted email folders, but you seem to be saying that TB 102 still does not have a functional method for "recursing the tree" to open up each folder to "register" each folder in such a way that programs like QuickFolders can "see" all the folders" -- and in your case, it sounds like that function would also do the folder repairs you need.

(Just exposing/registering the folders is what I need. By exposing, I mean opening each folder that contains folders, not actually opening each terminal folder. That process seems to register in TB's database the folders-contained-in-the-folder-I-opened. However, It sounds like you need each folder, including terminal folders, to actually be opened and read (i.e. TB does a count of the number of emails in the folder).

I was not aware of corruption problems such as you have mentioned. That is very disturbing indeed. (If I had that problem, I would have more than 10,000 folders to repair.) Have there been other reports of such corruption? If not, then instead of thinking that TB may be causing the corruption, I would look closely at your user environment, the IMAP you are using -- if you are using IMAP -- and even the IMAP server's hardware itself (disk management and RAM) for errors. Have you have corruption of any files other than email files? If yes, then especially be looking at disk errors and RAM errors. (I suggest this from long-ago experience. Strange things can happen with intermittent RAM errors.)

Best of luck to you. If you find a solution, please share it with me!

Jay Smith