Global Search: Message picked in Facet view not selected with displayed in conversation, TB bug 1856266

[mayerwin]

It'd be amazing if you could help fix Global Search, which has been completely broken by Supernova as per this forum thread.

Easy to reproduce:

1. Search for something that is in a multi-email conversation with global search.
2. Pick a result.
3. You'll be taken to a list of ALL emails from the selected conversation, not the specific email from that conversation containing the snippet you clicked on.

It is quite infuriating as it was working fine before Supernova :(. Such poor QA.

Another thing that'd be great would be if you could deduplicate emails in the thread view, as with Gmail accounts for example, it'll always show the same email twice: once as part of the "Sent" folder, once as part of the "All mail" folder. It'd make sense to only show the email from the Sent folder if a duplicate (with exact same hash) is found with an email in "All mail". I'd donate $50 if this was fixed.

[Betterbird]

A few things here:

1. connect.mozilla.org is pretty much completely useless. A discussion club that is 99.995% ignored.
2. Please file an upstream bug at https://bugzilla.mozilla.org - not necessary, see below.
3. "Such poor QA" is not correct. TB has close to no QA whatsoever, the product "ripens" exclusively at the customer.

Note that BB has a pref gloda.list.immediate described at https://www.betterbird.eu/expert-tips/ to skip the facet view.

I tested this a bit in 102 and 115: The list view is actually the same with potentially a lot of messages, however, in 102 the relevant message is selected. So if I see it correctly, you're complaining about this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1856266

That but is on our watch list and we'll ship it a.s.a.p., if you wish, we can prepare a BB 115.4.0-bb16 "latest build" with the fix included. Which platform do you need? Windows, Mac, Linux?

Gmail is a total pain. We suggest to not subscribe to "All Mail".

[mayerwin]

Thanks for your super fast reply. Yes, it looks like the global search issue is filed in the bug you found: https://bugzilla.mozilla.org/show_bug.cgi?id=1856266. And if I understand correctly, they've fixed it 2 hours ago (but it is not yet rolled out).

For the duplicate email issues, we need to be able to search across archived emails, and unfortunately it seems there is no way to have a folder only containing archived emails that are not in "Sent". So we absolutely have to subscribe to the "All Mail" folder. A fix could be to "just" go over the loaded list of emails in the thread view, and hide those from "All Mail" if an identical email is found in "Sent" (maybe with a setting to enable or disable this behavior). Does it sound very challenging to achieve based on your knowledge of the related code?

[mayerwin]

ChatGPT might be able to give some ideas (worth what it's worth, probably can be steered to be more helpful with more precise guidance and API definitions it may not know about): https://chat.openai.com/share/70facdb0-5dcb-4e67-a19a-ad6549f4ce27

[Betterbird]

Yes, the bug was fixed on the daily repo a few hours ago. It will ship in ESR 115 in a few weeks. BB could make it happen sooner, but you didn't request it.

I believe that Gmail offers a horrible user experience. It's also not well supported in TB/BB. Furthermore, Google doesn't provide any customer service, they can lock your account at their discretion at any time. And what guarantee do you have that your e-mail doesn't disappear one day. We won't implement anything special for Gmail. Get a real account with a real paid provider with some real service following real protocols and not that crazy Gmail paradigm. Sorry.

[mayerwin]

I'd definitely appreciate if you could ship the bugfix into BB sooner, but I'll respect your normal release schedule :).

Yes, I'm not a fan of the Gmail user experience (the lack of innovation for such a massively used product, and horrible UX choices have been disappointing me for years), however please note that:

• Gmail has over 1.8 billion active users as of 2023. which means 22.22% of the world’s population uses Google’s mail service.
• Gmail has a 28% market share globally in the email management market.
• Gmail owns 75.78% of the market share in the United States. These figures come from https://www.demandsage.com/gmail-statistics/. Statistics vary based on the source, but in all cases the numbers are undeniably massive.
• It is not straightforward to migrate to a new email address if you've been using a Gmail address for years, considering how much email it is intertwined with your online identity (not a good thing, but that's what it is).
• Most importantly, Gmail is far from only being used for personal needs. There is a huge professional user base since it is also the email component of Google Workspace. Employees cannot decide on their own to use a different email service provider than what their company has chosen.

So while I agree it is annoying, supporting it as best as possible seems like a good idea to avoid losing a lot of users. Thunderbird did make good progress over the years to integrate the calendar, address book, and email management from Gmail. Outlook also eventually did this begrudgingly after having offered a really horrible UX for Gmail users for years to push companies towards Office 365. And it should be possible to take advantage of the complete flexibility of writing our own email clients to offer work arounds for most of Gmail's problems (that's one huge advantage of open source clients vs. Outlook which depends on Microsoft's goodwill and may never make the experience as good as for emails hosted on Office 365).

[Betterbird]

We will make a "latest build" in the next few days, well before 115.4.1 on the 6th of November.

A lot could be said about Gmail. Suffice to say that might isn't always right. If you never make the effort to leave and untangle the twines then, well, you will never leave. We believe that a paid mail provider maybe together with your own domain is the only professional solution. It costs less than a coffee per month. betterbird.eu costs 2€ per month all inclusive at the best hoster in Germany.

So if TB/BB don't support Gmail properly, they will lose (paying?) users to which application exactly which has global search and handles Gmail better?

All that said, we're still not convinced that subscribing to "All Mail" is necessary. If "All Mail" summarises all other mailboxes, why is there a need to subscribe to it? I don't understand (quote): we need to be able to search across archived emails, and unfortunately it seems there is no way to have a folder only containing archived emails that are not in "Sent". So we absolutely have to subscribe to the "All Mail" folder.

[mayerwin]

Thanks! It is true that you need to make an effort to untangle the twines, otherwise you'll never do it, but that's what happens :). Even if I end up doing it personally, most other users won't do it. And for most business needs, Google Workspace (which costs a bit more than 2€/month/user) is still a decent, comprehensive solution despite its numerous flaws.

If TB/BB don't support Gmail properly, users may begrudgingly end up going back to Gmail.com itself :). This is what I often end up doing when Global Search fails (but that was before I discovered Betterbird, looking forward to testing the complex search terms feature).

All that said, we're still not convinced that subscribing to "All Mail" is necessary. If "All Mail" summarises all other mailboxes, why is there a need to subscribe to it? I don't understand (quote): we need to be able to search across archived emails, and unfortunately it seems there is no way to have a folder only containing archived emails that are not in "Sent". So we absolutely have to subscribe to the "All Mail" folder.

Actually, All Mail does not summarises all other mailboxes/folders, it also contains emails which may be nowhere else: when you "archive" an email from your inbox (to reach "inbox zero"), it ends up in no folder at all, except "All Mail" (showing up within "Archives" if you enable "Unified folders" in Thunderbird). So if you don't subscribe to "All Mail", you won't be able to search for that particular email. Let me know if this is still not clear.

It is annoying as it means a lot more storage than necessary is being used (all the emails still in the inbox, in any label, and in sent will also be duplicated in "All Mail"). I don't understand why Gmail hasn't simply created an "Archived" folder for this purpose (for emails that were in the inbox only), but that's how it works.

A more general (still Gmail-specific) solution to the problem would be to have Thunderbird create a virtual "Unlabelled" folder that would only contain emails from "All Mail" that don't have any other label. These are the only ones we want to see, as it'll naturally avoid including emails in "sent", "inbox", or other folders/labels. Right now, there is no way to see only emails archived from "inbox", so it'd finally make this possible. Gmail kinda supports this with the following search query: has:nouserlabels -label:inbox -label:sent -label:chats -label:draft -label:spam -label:trash I had actually posted an article about this 10 years ago when the has:nouserlabels operator was rolled out... :)

If the labels are showing up somewhere when querying the list of email headers from "All Mail", that'd be the most efficient to apply this filtering early on and avoid downloading all the email bodies unnecessarily. Otherwise it'll for sure be quite an undertaking (then it might be practical to just aim to avoid showing duplicates in conversation thread views, this should be much simpler).

[Betterbird]

Horrible paradigm.

... except "All Mail" (showing up within "Archives" if you enable "Unified folders" in Thunderbird). So if you don't subscribe to "All Mail", you won't be able to search for that particular email. Let me know if this is still not clear.

This is not clear. "Unified folders" just groups folders in the folder tree differently. If the mail is in the Archives folder, unified or not, then a search should find it and there should be a need for "All Folders".

Has a bug been filed for this at TB's bugtracker Bugzilla? We're also 100% off topic in this issue which is about Gloda search.

[Betterbird]

This fixes the Gloda selection issue as far as I could see in a test with two messages:

https://www.betterbird.eu/downloads/WindowsInstaller/betterbird-115.4.0-bb16-latest-build.en-US.win64.installer.exe

[mayerwin]

Wow you rock! It works flawlessly now. Let me file another issue in GitHub for the handling of archived emails and try to clarify it a bit better, as you're right it is 100% off topic. No bug has been filed for this in Bugzilla yet.

[Betterbird]

Wow you rock! It works flawlessly now.

It's not our fix, we're just advancing it.

Let me file another issue in GitHub for the handling of archived emails and try to clarify it a bit better, as you're right it is 100% off topic. No bug has been filed for this in Bugzilla yet.

It needs a bug in Bugzilla.

[mayerwin]

OK so I ended up writing an add-on for this (my first one! Took me a few hours to figure everything out): https://github.com/mayerwin/thunderbird-hide-all-mail-duplicates

Would probably deserve to be merged to the core, but only if improved to only apply to Gmail and Google Workspace, for all languages as 'All Mail' might change based on the display language configured in Gmail), and maybe optimize the performance a bit when checking for duplicates (the aItems email collection could be projected once into a dictionary of emails indexed by messageId).

The code to patch is in https://github.com/mozilla/releases-comm-central/blob/master/mailnews/db/gloda/modules/GlodaSyntheticView.jsm and should be updated using the following logic (instead of monkey patching the prototype):

GlodaSyntheticView.prototype.reportResults = function(aItems) {
    try {
        // Filter out duplicate messages from aItems before calling the original function
        const filteredItems = aItems.filter(item => {
            const hdr = item.folderMessage;
            if (hdr) {
                // Determine if an item is a duplicate
                //console.log("Email13: " + hdr.messageId + " in " + hdr.folder.name);
                if (hdr.folder.name === "All Mail") {
                    // Check the rest of aItems for a message with the same messageId but a different folder
                    const duplicateItem  = aItems.find(otherItem => {
                        const otherHdr = otherItem.folderMessage;
                        return otherHdr &&
                               otherHdr.messageId === hdr.messageId &&
                               otherHdr.folder.name !== "All Mail";
                    });
                    // If it's a duplicate, filter it out by returning false, but make sure to update selectedMessage if necessary.
                    if (duplicateItem && hdr.messageId === this.selectedMessage.messageId) {
                        this.selectedMessage = duplicateItem.folderMessage;
                    }
                    return !duplicateItem ;
                }
            }
            return true;
        });

        // Call the original reportResults function with the filtered list of items
        GlodaSyntheticView.prototype.originalReportResultsGlobal88319.call(this, filteredItems);
    } catch (e) {
        console.error('An error occurred while filtering items:', e);
        return true;
    }
    //console.log(name);
    //Services.wm.getMostRecentWindow("mail:3pane").alert(name);
}

If you think it'd be a nice feature in BetterBird, feel free to add it once it matches your quality bar. Definitely must be a pain point for all Gmail/Google Workspace users.

However, a better fix would be to prevent downloading emails in All Mail in the first place if they are already present in another folder. But this sounds a bit more complex to achieve, as we can't know if the emails are present in another folder until we've checked all folders. The email headers should suffice as the message ID is identical so it should help with the performance, but still it isn't trivial to implement it optimally to not slow things down, and Thunderbird may not have ready logic to use to avoid loading some emails (and hiding them without deleting them) within a folder.