Chart Generation Taking a Long Time (or not happening at all)

[ryanleesipes]

Describe the bug Generating charts for my inbox seem to take a very long time (I have yet to see the charts actually display). Have waited for charts to appear, assuming it is because of my large inbox. But have waited for likely an hour with the tab open at this point and still no charts. The spinner is spinning so I assume something is happening, but I'm not sure if I should continue to wait or if it's a bug,

To Reproduce

1. I click on the button to display the stats dashboard.
2. The top-line stats display, such as: Mails total, Mails unread, Mails received, etc
3. None of the charts display information. Just black where the charts should show up, the titles however are there.

Expected behavior Expected the charts to appear and update as more information was processed, or the charts to show up faster.

Screenshots

System (please complete the following information):

• OS: elementaryOS 5.1.7
• Thunderbird Version 78.3.1 (64-bit)

[devmount]

This definitely is a bug. I have an inbox with ~30k emails which takes 13s to generate the charts. This is still quite long, so performance should definitely be improved, but never finishing chart generation is a bug.

Do you have multiple email accounts? Does the popup loading spinner finish (before clicking your account and opening the stats page)?

[ryanleesipes]

Just one account.

The spinner does not seem to stop if I let it sit before opening the page (in the pop over menu).

[devmount]

Ok, thank you for your feedback! I'll try to reproduce this and come back as soon as I have a solution.

[ryanleesipes]

I can look at the console or anywhere you think there might be useful information.

[devmount]

I set up a virtual environment with elementaryOS 5.1.7 and Thunderbird Version 78.3.1 (64-bit) and wasn't able to reproduce your issue. So I assume it must be something with your emails or folder structure.

Can you please do the following:

• open the console (add-ons > gear > debug add-ons > scroll to ThirdStats > Inspect)
• clear console (small trash icon on the upper left)
• open (or right click > reload if its already open) the Stats tab
• make a screenshot or copy any errors/warnings and paste it here

Thank you!

[devmount]

Plus: I just released v0.4.0 with improved loading indication (you shouldn't see empty charts anymore now). Maybe this changes anything for you, @ryanleesipes?

[ryanleesipes]

Here is the screenshot you requested.

I do like the new indicator:

[devmount]

Thank you for the screenshot! I created a possible fix. To test it, please do the following:

1. Clone this repo git clone https://github.com/devmount/third-stats
2. Change into the created directory cd third-stats
3. Switch branch git checkout fix-infinite-loading
4. Open Thunderbird and uninstall ThirdStats, if you already installed it
5. Open Thunderbird > main menu > add-ons > gear menu > debug add-ons > load temporary add-on > choose the manifest file in the cloned repo's directory
6. Check, if charts are loaded now. If not, please create a screenshot of the console again.

Thank you very much @ryanleesipes for taking the time to test and debug this issue.

[ryanleesipes]

So two things to note (one might be the issue).

I installed Thunderbird on this machine via Flatpak so I wonder if that is causing issues. When attempting to load the temporary add-on I get the following message:

Extension is invalid Loading locale file _locales/en/messages.json: Error: Error while loading 'file:///run/user/1000/doc/c28881a6/_locales/en/messages.json' (NS_ERROR_FILE_NOT_FOUND)

This could be because Flatpaks have their own filesystem, I'm troubleshooting.

[devmount]

Interesting, is that file actually existing on that location? Looks like the root directory of ThirdStats was not properly recognized... I don't have any experience with Flatpak, so thank you for troubleshooting!

[koobs]

I'm seeing the same issue on:

• Windows: 10 Pro x64 (2004, 19041.572)
• Thunderbird: 82.0b3 (64-bit) (Installed via AMO)
• Third-Stats: 0.5.0

Additional symptoms to report:

• The processing hangs after the same number of messages are parsed. In my case: 82,562 (now 82,564 after 2 new messages are received)
• I see the same errors in debug addons console output as @ryanleesipes

However, when I view the standard console output (not via debug addons), I see some additional errors that dont appear to show up during an 'addon debug' context. This is with the full-page third-stats loading, not just stats via the menu icon.

Full Console Log: console-export-2020-10-20_21-33-0.txt

Looking at the process stack (process explorer, see screenshot), the only potentially interesting thing I can see is en elevated number of calls to RegExprParser::* and configthreadlocale activity after the mail parsing process hangs, which continues until closing the third-stats tab. The number of calls to TargetNTUnmapViewofSection is at least two orders of magnitude higher too (baseline is <= 0.03 CPU when the issue is visible)

Additional information. Unsure if relevent or related:

• The processing (count) seems to restart at zero each time the toolbar icon is opened -> closed -> opened (may be intended)

[devmount]

Thank you @koobs for this detailed description and error analysis! 👏🏻 This definitely helps. Have you checked the possible fix I created? Does it change anything?

Thanks to your error log I can do some additional tests and see if I can find what's causing this issue.

The processing (count) seems to restart at zero each time the toolbar icon is opened -> closed -> opened (may be intended)

A click of the toolbar icon always reloads the popup content, so the counts are processed again - I can see if it's possible to implement a storage for already processed email/folder counts. If this is a feature you would like, you can create a feature request for that.

[tonewarper]

I use TB 78.4 on MacOS Catalina 10.15.7 … and experiencing the exact same behaviors. You’re seeing this on 3 OS’s , so it does not seem OS-related.

[devmount]

Thank you @tonewarper for confirming this issue. Unfortunately I cannot reproduce this on my own machines, so I'm waiting for someone to test the possible fix. I can provide the fix as XPI file, if that helps testing it.

[devmount]

Next version of ThirdStats contains a possible (!) fix, let's see if this issue persists.

[devmount]

@tonewarper @koobs @ryanleesipes The possible fix is now public in the Thunderbird add-on repository. Please update ThirdStats to v0.7.0 and let me know, if you still have this issue. Thank you all for testing and improving this add-on!

[koobs]

@devmount Success! Thank you 👍 Now we can focus on overwhelming you with feature requests ;D

Which commit was the likely fix for the issue? (Edit: Im guessing https://github.com/devmount/third-stats/commit/97506847a23cd20c9091068f124af0046ada24e7 ?)

[devmount]

Great! I'm very happy that it works for you now 👏🏻

Now we can focus on overwhelming you with feature requests ;D

Go ahead, I'm looking forward to your ideas 💡👍🏻😅

Yes, it was 97506847a23cd20c9091068f124af0046ada24e7

[tonewarper]

I am the bearer of unpleasant news… on my end, this very-same behavior still persists on v0.70 , even after restarting TB twice. Perhaps it worked for @koobs on Windows while I’m on MacOS? Or maybe there is another variable at play. Most of my mails are fetched via IMAP on Gmail servers. It’s peculiar that Thunderbird Stats is able to produce a count of the mails but doesn’t generate the charting. The counter counts up while calculating, and at some point the counting stops while the wheel keeps spinning and the ‘Fetching’ loader remains where the charts should be… forever.

[devmount]

@tonewarper Sorry that it didn't work for you. Can you share your console output (as described here)?

[devmount]

So for @koobs is the fix working, for @tonewarper not. How about you @ryanleesipes? Did 0.7.0 fix this issue for you? @tonewarper Did you have some time to look into the console?

[6Sim9]

Hi, Same issue here: TB 78.4.3 (64b) / W10

It works great with my local folder...

but fail with my main account:

I hope it helps...

[devmount]

Hi @6Sim9, thank you for these valuable additional information. It still looks like an issue with the folder processing, so I implemented an error handler to provide more information and gracefully continue processing in case of failure (d3203fd09fe6d5fea76d4fd9088eceed741641db). It will be part of the next release v0.8.0 - let's hope that it works 🙏🏻 (I still can't reproduce this issue on any of my machines)

[devmount]

ThirdStats v0.8.0 is public. @6Sim9 @tonewarper: Please upgrade your ThirdStats add-on and check if the problem still persists. Have a nice weekend.

[tonewarper]

Updated and restarted, bug still alive. There are several CallResult for closed conduit Child errors when I switch to the add-on but I think here is when — yes — I also experience folder processing issues:

[devmount]

Sorry to hear that. I'll further investigate this issue and will let you know when I have a solution.

[6Sim9]

Hi! Same here :'(

I've think about something... Maybe it's a clue. My mails are stored on a different drive letter(E:) than TB (C:).

Have a nice week :)

[devmount]

Thank you for confirming this issue. Before I make another guess of a solution, we need to debug this issue on your end, to really find the cause of the problem.

@6Sim9 and @tonewarper Can you please do the following:

1. Open Thunderbird
2. Open the ThirdStats stats page and wait ca. 1min
3. Open the console (main menu > add-ons > gear > debug add-ons > scroll to ThirdStats > Inspect)
4. Clear console (small trash icon on the upper left)
5. Past the following snippet and hit the Enter key:

   let accounts = await messenger.accounts.list(); let allfolders = [];
   accounts.map(a => { function traverse(folders) { if (!folders) return; for (let f of folders) { allfolders.push(f); traverse(f.subFolders); }} traverse(a.folders);})
   console.log(allfolders);

   This should print a list of all folders of all accounts or - most probably in your case - a hopefully helpful error message.

6. See if you can see all folders and paste error messages here

[6Sim9]

Hi! 222 folders found but no error.

[devmount]

Interesting. If you expand the folder list, are all folder entries valid or are there folders without a name or without a path?

[6Sim9]

All seems to be OK. Here is an exemple:

[devmount]

Ok, now that we know, that the folder list is correct, let's take it a step further and count all messages of all folders of all accounts. Please enter the following, if you have the console still open:

let count = 0;
await Promise.all(allfolders.map(async f => { let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } })).then(() => { console.log(count); });

If you have a new console, here is the full code snippet:

let accounts = await messenger.accounts.list(); let allfolders = []; let count = 0;
accounts.map(a => { function traverse(folders) { if (!folders) return; for (let f of folders) { allfolders.push(f); traverse(f.subFolders); }} traverse(a.folders);})
await Promise.all(allfolders.map(async f => { let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } })).then(() => { console.log(count); });

The output can take some time, according to the number of messages you have. Please let me know, if you now have any error messages.

[tonewarper]

I followed the same routine you asked of @6Sim9 . Message counts not working -

folder is null ext-messages.js:189 list chrome://messenger/content/parent/ext-messages.js:189 list self-hosted:844 result resource://gre/modules/ExtensionParent.jsm:996 withPendingBrowser resource://gre/modules/ExtensionParent.jsm:602 result resource://gre/modules/ExtensionParent.jsm:996 callAndLog resource://gre/modules/ExtensionParent.jsm:958 recvAPICall resource://gre/modules/ExtensionParent.jsm:995 AsyncFunctionNext self-hosted:693 Uncaught (in promise) Error: An unexpected error occurred

Folder name/path fetching worked without any errors, same as @6Sim9 . Out of 65 folders for me, all look quite normal except for one: ( "?", path: "/", … } ) which is how it appears in TB, perhaps a TB bug in fetching that foldername from Gmail with IMAP or not. The rest look quite ordinary, just some have [ ] or spaces or @ characters which I don’t think is a problem since they’re inside the “ “. .

At first given this error being so agnostic to OS I thought it wasn’t an OS issue, but one must also wonder if there is something to do with the way Mac’s FS or permissions work.

[devmount]

Thank you @tonewarper for these information. The code snippet above basically uses the same method given as example in the docs: https://thunderbird-webextensions.readthedocs.io/en/78/how-to/messageLists.html

Due to resource management message lists are splitted into pages which have to be iterated over to get all messages of a folder. So seeing how this error appears, I assume that it's caused by a bad combination of OS file handling, Gmail folders and the way, Thunderbird handles that. As I don't see a way to solve this in ThirdStats, I issued a corresponding topic at Thunderbirds topicbox.

Let's hope, that somebody knows a solution 🙏🏻

[6Sim9]

Hi! Here are my results:

I don't have any IMAP nor Gmail account.

[devmount]

Thank you @6Sim9 for your results and these additional information. So when you're not using IMAP, you use good ol' POP3? I see if someone at Thunderbirds topicbox can help us here.

[arndissler]

Hi there, I'm currently not able to reproduce it (yet), but I have found the problem in the Thunderbird source code where the error occurs and will investigate further. As far as I can see there's no bug ticket filed on Bugzilla, so I collect more information and file one.

The main problem so far is to find a reliable scenario so that one is able to test and reproduce the error and to ensure that it definitively was fixed. To find this scenario it would be helpful to have some more information at hand, e.g. what type of accounts are set up, what folders are available in this account and maybe some more details about the server side like if there any shared IMAP folders, and the profile itself, e.g. are there any disabled accounts or lately deleted folders.

@6Sim9 & @tonewarper are you available for giving more insights about your setup and/or executing some more scripts and share the information with me? It might collect personal information like account and folder names, so this thread is may not be the perfect match to share this sensitive information.

I'll keep you updated.

[devmount]

Thank you @arndissler for investigating this issue further 👏🏻

[devmount]

It looks like it could be caused by this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1529791

@6Sim9 & @tonewarper can you somehow determine if you use virtual folders?

[koobs]

@devmount For what its worth, I'm running (successfully, without errors) ThirdStats on a Gmail backed account too and not running any special settings or folder subscribe settings. I mention this as "[Gmail]" is mentioned in the the bug mentioned in the previous comment. See screenshots below:

You can also see a "MFH Pending" folder (a 'Saved Search'), which might also be classed a virtual folder (i'm not sure)

[devmount]

@koobs thank you. Another user just confirmed, that he is successfully using ThirdStats with virtual folders, so that might not be the problem here. I'll take a look what other folder types exist and if there is any type currently not supported by the webextension API.

[arndissler]

Is anyone who's facing this issue here up for testing a "daily build" of Thunderbird? The testing procedure will be as follows:

1. clone (duplicate) your Thunderbird profile where the error occurs
2. download a specific Daily build of Thunderbird
3. extract the ZIP (Windows) or open the DMG (macOS)
4. start the Thunderbird from the ZIP/DMG via use the cloned profile from step 1., so Thunderbird should be started from the command line with the "profile" option set
5. open the Debugger as described in this comment, step 1. to 4.
6. run the scripts which are provided by devmount in this comment
7. there should be still errors in the console, but they should mention the exact folder, e.g. Error listing messages in folder 'your-foldername-here' in account 'account-name'

The Daily build containing the patch are available for Windows 32bit, Windows 64bit and macOS

To start Thunderbird Daily with the cloned profile, run Thunderbird.exe --profile C:\path\to\profile on windows or /Volumes/Thunderbird\ Daily/Thunderbird\ Daily.app/Contents/MacOS/thunderbird --profile /path/to/profile on macOS, ensure to modify the pathnames according to your local system.

[koobs]

@arndissler Probably not the best place for this, but relevent nonetheless given the request. If I (and users) had control over Telemetry (the pref is locked True on beta/nightlies), we'd use nightly (and I use to). I'd be more than happy and in fact, I'd prefer, to run nightly again, particularly to help support extension developers ramp up after the mass webextensions exodus

Edit: Thank you for the bug fix / patch upstream :)

[tonewarper]

I’m not sure if I use virtual folders, but what I do know for sure is that I’ve configured my IMAP folders in TB to sync all Gmails and store on my hard drive too for faster search-speed, so I have a massive profile size (I still use PST/Mbox not Maildir, if that makes a difference). I can also now confirm that the “?” folder path I mentioned earlier was indeed not a TB bug implicated in the null result… it was relabelled that way inside the Gmail interface.

While I’d love to dedicate to troubleshooting and deep-testing, what I’m able to commit to is occasional quick validation/verification over the long haul. I’ll keep checking in on this project! I’m happy that at least you’ve gotten to some degree of pinpointing the problem for some users like me, and hope that in time more users experiencing it will chime in here to get us over to the promised land.

[arndissler]

Four your information: it seems like the fix for this behaviour will be available with Thunderbird 85.

[devmount]

Thank you Arnd 👏🏻

[6Sim9]

Hi! I don't know if it's help. I copied my email folder from d: to c: and created a new account with this local directory.

And it works!

Initial account still doesn't work...

[devmount]

I copied my email folder from d: to c: and created a new account with this local directory.

This confirms, that it's a problem with IMAP folders. I guess that converting the IMAP account to a local account creates "real" folders only which the WebExtensions API can handle properly. We will wait for Thunderbird 85 to be released to test this issue again.

[6Sim9]

OK ... I'm sorry ... I realize I didn't respond when you asked me about IMAP or POP. My server setting is "POP mail server".

[devmount]

No problem. It seems to be the same for POP mail server. There are special folder types on your mail server Thunderbird can't handle properly. Let's hope, that TB85 gets released soon.