Quick filter issue on virtual folder with complex search term

[viking2]

I have created a complex filter:

After I have read a message from "Groupon", "Swagbucks" or "Linkedin", I do not want it to show up in the message pane anymore. This works great (note that I need to select another folder and then come back for the read messages to disappear)!

I also do not want messages from xyz to show up at all unless the subject contains "you have been". That is the purpose of the last two lines. This works fine as well.

However, when I perform a quick search for "Groupon" it also finds "Swagbucks" and "Linkedin". If I remove the last two lines in the complex filter, a quick search on "Groupon" only finds "Groupon" as it should.

Why do the last two lines affect the quick search? Is it a bug or am I missing something?

[Betterbird]

(note that I need to select another folder and then come back for the read messages to disappear)

Already discussed here: https://forums.mozillazine.org/viewtopic.php?p=14985872 Amazing, you ask the same question again and get the same answer.

As for the other question: Since you're searching a virtual folder that is defined by a rather complex expression, it's possible that joining the quick filter terms creates an incorrect expression.

To diagnose this, the system actually dumps out the resulting search term to the console. Please submit this information.

Example: Condition: AND [from,contains,huhu) OR (to or cc,contains,huhu) OR (subject,contains,huhu] [QuickFilterManager.sys.mjs:624:15](resource:///modules/QuickFilterManager.sys.mjs)

However, that only shows the part generated by the QF, we would have to ship you a special version to dump out the entire term. In the meantime, can you provide the definition of the search term from virtualFolders.dat.

[viking2]

(note that I need to select another folder and then come back for the read messages to disappear)

Already discussed here: https://forums.mozillazine.org/viewtopic.php?p=14985872 Amazing, you ask the same question again and get the same answer.

Actually, it was not a question. I only mentioned it in case someone wanted to try the complex filter for themselves.

As for the other question: Since you're searching a virtual folder that is defined by a rather complex expression, it's possible that joining the quick filter terms creates an incorrect expression.

To diagnose this, the system actually dumps out the resulting search term to the console. Please submit this information.

Example: Condition: AND [from,contains,huhu) OR (to or cc,contains,huhu) OR (subject,contains,huhu] [QuickFilterManager.sys.mjs:624:15](resource:///modules/QuickFilterManager.sys.mjs)

Here is the search term from the console: Condition: AND [from,contains,groupon] QuickFilterManager.jsm:605:15

However, that only shows the part generated by the QF, we would have to ship you a special version to dump out the entire term. In the meantime, can you provide the definition of the search term from virtualFolders.dat.

Here is the search Term definition from virtualFolders.dat: terms=OR [date,is after,01-Jan-2024) AND (status,isn't,read) AND [from,contains,Groupon) OR (from,contains,Linkedin) OR (from,contains,Swagbucks]] OR [date,is after,01-Jan-2024) AND (from,doesn't contain,Groupon) AND (from,doesn't contain,Linkedin) AND (from,doesn't contain,Swagbucks) AND [from,doesn't contain,xyz) OR (subject,contains,you have been]] searchOnline=false

[Betterbird]

Can you supply the data/folder the virtual folder is based on. Otherwise it will be too laborious to craft the data to execute the query on. Or any other stripped down version that will demonstrate the issue.

[viking2]

Can you supply the data/folder the virtual folder is based on. Otherwise it will be too laborious to craft the data to execute the query on. Or any other stripped down version that will demonstrate the issue.

Do you mean all the messages in the folders? The virtual folder pulls messages from 2 different inboxes (Total 261,402 messages) which results in 18,674 messages.

[Betterbird]

Any data that will reproduce the issue. Since this appears to be a rule evaluation issue, one folder with 10 messages, a mix of read/unread and stuff from Groupon, Linkedin, Swagbucks, xyz and a few with(out) subject "you have been" should be sufficient.

[viking2]

Any data that will reproduce the issue. Since this appears to be a rule evaluation issue, one folder with 10 messages, a mix of read/unread and stuff from Groupon, Linkedin, Swagbucks, xyz and a few with(out) subject "you have been" should be sufficient.

OK. May take some time to figure out... How do I prepare the folder with the messages to search on? Do I create a local folder and copy the messages there? What files do I send you?

[Betterbird]

We received a data sample that reproduces the issue. In fact, with the last two search terms, QF doesn't appear to have any effect, as looking for "huhu" just shows the content of the search folder unaltered:

When the last two terms in the search folder definition are removed, QF works as it should. We'll debug the issue. At a guess, the additional QF condition is ORed to the existing term leaving the result unaltered.

[Betterbird]

The virtual folder definitions communicated to us were: Non-working: terms=OR [status,isn't,read) AND [from,contains,Groupon) OR (from,contains,Linkedin) OR (from,contains,Swagbucks]] OR [from,doesn't contain,Groupon) AND (from,doesn't contain,Linkedin) AND (from,doesn't contain,Swagbucks) AND [from,doesn't contain,xyz) OR (subject,contains,you have]]

Working: terms=OR [status,isn't,read) AND [from,contains,Groupon) OR (from,contains,Linkedin) OR (from,contains,Swagbucks]] OR [from,doesn't contain,Groupon) AND (from,doesn't contain,Linkedin) AND (from,doesn't contain,Swagbucks]

To this we need to append the QF condition AND (from,contains,Groupon). Due to limitations in the term evaluation, an overall OR group becomes problematic when an AND term is appended. This needs further analysis.

[Betterbird]

It's possible that appending AND (from,contains,Groupon) to the double group end ... AND [from,doesn't contain,xyz) OR (subject,contains,you have]] causes an issue. We'll investigate.

[Betterbird]

We looked at the expression trees which are built. First the good example. We'll make the different tree structure when AND (from,contains,huhu) is added:

===  op: AND
  === rite op: OR
    === left op: AND
      === left op: AND status,isn't,read
      === rite op: OR
        === left op: OR
          === left op: AND from,contains,Groupon
          === rite op: AND from,contains,Linkedin
        === rite op: AND from,contains,Swagbucks
    === rite op: AND
      === left op: AND
        === left op: AND from,doesn't contain,Groupon
        === rite op: AND from,doesn't contain,Linkedin
      === rite op: AND from,doesn't contain,Swagbucks

===  op: AND
  === rite op: AND <-- additional AND here
    === left op: OR
      === left op: AND
        === left op: AND status,isn't,read
        === rite op: OR
          === left op: OR
            === left op: AND from,contains,Groupon
            === rite op: AND from,contains,Linkedin
          === rite op: AND from,contains,Swagbucks
      === rite op: AND
        === left op: AND
          === left op: AND from,doesn't contain,Groupon
          === rite op: AND from,doesn't contain,Linkedin
        === rite op: AND from,doesn't contain,Swagbucks
    === rite op: AND from,contains,huhu <-- additinal AND here.

[Betterbird]

Not the bad case:

===  op: AND
  === rite op: OR
    === left op: AND
      === left op: AND status,isn't,read
      === rite op: OR
        === left op: OR
          === left op: AND from,contains,Groupon
          === rite op: AND from,contains,Linkedin
        === rite op: AND from,contains,Swagbucks
    === rite op: AND
      === left op: AND
        === left op: AND
          === left op: AND from,doesn't contain,Groupon
          === rite op: AND from,doesn't contain,Linkedin
        === rite op: AND from,doesn't contain,Swagbucks
      === rite op: OR
        === left op: AND from,doesn't contain,xyz
        === rite op: AND subject,contains,you have

===  op: AND
  === rite op: OR
    === left op: AND
      === left op: AND status,isn't,read
      === rite op: OR
        === left op: OR
          === left op: AND from,contains,Groupon
          === rite op: AND from,contains,Linkedin
        === rite op: AND from,contains,Swagbucks
    === rite op: AND <-- additional AND here
      === left op: AND
        === left op: AND
          === left op: AND
            === left op: AND from,doesn't contain,Groupon
            === rite op: AND from,doesn't contain,Linkedin
          === rite op: AND from,doesn't contain,Swagbucks
        === rite op: OR
          === left op: AND from,doesn't contain,xyz
          === rite op: AND subject,contains,you have
      === rite op: AND from,contains,huhu <-- additional AND here

It appears that the additional term got glued into an incorrect position in the tree. To be continued.

[Betterbird]

Found the issue. Double group endings only rewind the stack by one, when they should rewind by as many terms as end there. So the additional branch gets added to low in the tree. We need to think about a solution now.

[Betterbird]

WIP. (obsolete attachment deleted)

[viking2]

Is there a new BB version that incorporates this fix? I have other issues with randomly disappearing messages and I need to keep updating the Saved Search to get them back. My Saved Search contains complex filters. I would like to test a new BB version with improved filtering before attempting to report.

[Betterbird]

WIP = work in progress. It's highly complex and not fixed yet. No ETA.

[viking2]

(Comment removed for presenting three unrelated issues.)

[Betterbird]

You need to file separate tickets for these issues. In this ticket, we will fix what you reported in the first comment. That said, your point 1 is unclear, please clarify when you file a new ticket. We will proceed to deleting your previous comment as it t is off-topic in this ticket, so please take a copy.

[Betterbird]

Try: https://www.betterbird.eu/downloads/WindowsInstaller/betterbird-115.17.0-bb35-latest-build4.en-US.win64.installer.exe

Using the data from https://github.com/Betterbird/thunderbird-patches/issues/365#issuecomment-2467925985 and adding a QF with Groupon, we now get the three desired messages instead of five. That's the only test we've done.

[viking2]

Thanks, but it doesn't work for me. Using QF with Groupon as sender still shows all five messages. Am I downloading the correct version: "betterbird-115.17.0-bb35-latest-build4.en-US.win64.installer"?

[Betterbird]

Yes, build4 contains the fix. Works here. Content of virtualFolders.dat: terms=OR [status,isn't,read) AND [from,contains,Groupon) OR (from,contains,Linkedin) OR (from,contains,Swagbucks]] OR [from,doesn't contain,Groupon) AND (from,doesn't contain,Linkedin) AND (from,doesn't contain,Swagbucks) AND [from,doesn't contain,xyz) OR (subject,contains,you have]].

Without QF:

With QF:

[Betterbird]

Decision tree without QF:

=== 00000269895A1940:  op: OR
  === 0000026989575760: left op: AND
    === 0000026989575730: rite op: AND
      === 00000269895757F0: left term: status,isn't,read
      === 00000269895A1A00: rite op: OR
        === 00000269895A1A90: left op: OR
          === 00000269895757C0: left term: from,contains,Groupon
          === 0000026989575CD0: rite term: from,contains,Linkedin
        === 00000269895A1AF0: rite term: from,contains,Swagbucks
  === 00000269895A1A60: rite op: AND
    === 00000269895A1C10: left op: AND
      === 00000269895A1970: left op: AND
        === 00000269895A1730: left term: from,doesn't contain,Groupon
        === 00000269895A1B50: rite term: from,doesn't contain,Linkedin
      === 00000269895A1BE0: rite term: from,doesn't contain,Swagbucks
    === 00000269895A1C40: rite op: OR
      === 00000269895A1AC0: left term: from,doesn't contain,xyz
      === 00000269895A1B80: rite term: subject,contains,you have

Decision tree with QF:

=== 0000026980235520:  op: AND
  === 00000269800BAD60: left op: OR
    === 000002697F336070: left op: AND
      === 000002697F336760: rite op: AND
        === 000002697F2EFDF0: left term: status,isn't,read
        === 00000269800BA8E0: rite op: OR
          === 000002697F3367F0: left op: OR
            === 000002697F336AF0: left term: from,contains,Groupon
            === 000002697F3368B0: rite term: from,contains,Linkedin
          === 000002697F3369A0: rite term: from,contains,Swagbucks
    === 0000026980235220: rite op: AND
      === 0000026980235100: left op: AND
        === 00000269800BAF70: left op: AND
          === 00000269800BADF0: left term: from,doesn't contain,Groupon
          === 00000269800BAEE0: rite term: from,doesn't contain,Linkedin
        === 00000269802350D0: rite term: from,doesn't contain,Swagbucks
      === 00000269802352E0: rite op: OR
        === 00000269802354F0: left term: from,doesn't contain,xyz
        === 0000026980235160: rite term: subject,contains,you have
  === 000002697F3366D0: rite term: from,contains,ExtraQF.

And that is correct. The ExtraQF term gets added at the end and another AND node is added at the top of the tree.

[viking2]

My mistake. I have a lot BB portable installations and a lot of different cores. I got them mixed up. It works great; QF for groupon => 3 messages; QF for Swagbucks => 2 messages. Thank You!

If I now just could solve the other serious issue of disappearing messages...

[Betterbird]

We'd appreciate if you could actually use the version we supply. If at first attempt you come to different results, double-check before raising further complaints. You're wasting developer time that gets taken away from other issues and additionally discourages the developers: Why fix it if the user still claims it doesn't work? :cry:

[viking2]

I agree, and I am really sorry that I wasted developer time to confirm, I double checked and triple checked that I used the correct version of the core. However, I had applied the core to the wrong version of the BB portable installation.