Feature Request: Filters should be able to run automatically outside Inbox

[ThiloteE]

Dear honorable RealRaven2000,

for years i have used your filter addon and once i am done with being a student and earn some money myself, i will give you a donation for sure. Thank you for your hard work and willingness to respond to the users of your addon. I am not sure if there is a way how to adress the following two problems, maybe there is a solution out there, maybe it can't be done, but the following is what annoys me the most in Thunderbird:

Specs: Windows 10 x64, Thunderbird 68, latest official version of quickFilters.

Description: Second level filters can only be used manually on specific folders. When having many filters that filter for subject / content, setting them to run automatically interfers with other filters. Apparently there also seems to be something wrong with the order of how filters are applied as they seem to try to run at the same time.

Detailed Description: Step 1) Receive e-mail into Inbox (Imap-Account, Yahoo.com) Step 2) (First level) Filter "jobs" filters all e-mails with subject "jobs" automatically from the inbox into the folder "jobs" Step 3) (Second level) Filter "International Relations jobs" filters all jobs with the content OR subject "International Relations" from the folder "jobs" into Folder "International Relations jobs".

--> Step 3 only works if used manually on the folder "jobs".

Desired Outcome: Step 3 should be able to run automatically and on a specific folder, not only the inbox Cause?: All filters run automatically on the inbox. When one selects a filter to run on a specific folder, but then closes the filter-menu and opens it up again, the filters are reset to run on the inbox. Furthermore there is no option when creating a filter to run on a different folder other than the inbox.

Greetings,

Thilo

[RealRaven2000]

Desired Outcome: Step 3 should be able to run automatically and on a specific folder, not only the inbox Cause?: All filters run automatically on the inbox. When one selects a filter to run on a specific folder, but then closes the filter-menu and opens it up again, the filters are reset to run on the inbox. Furthermore there is no option when creating a filter to run on a different folder other than the inbox.

Well and that one is hardcoded into the architecture of Thunderbird; I think we would move easier towards that goal if this was an additional action / option within the filter itself, e.g. something along the lines of

Implementation wise, this would have to set up a folder listener based on this folder action, so the Add-on would have scan all filters on Thunderbird startup, then set up listeners and try to apply these filters to new incoming mails into these folders. Obviously not something Thunderbird itself supports out of the box, but at least it would be a workaround to store this as a custom action (and it would probably need to be implemented by FiltaQuilla, which has more low level access to filtering than quickFilters)

At the moment - with the current functionality - I would implement this filtering behavior by having the second level filter above the first level one, so the the finer granularity ones are ran before the more general ones, for example:

with the first filter looking for specific subject line and body conditions and the second one using keywords in the subject line only.

[ThiloteE]

Your suggestion to create an additional action / option in filtaquilla seems fine :-) Obviously not the best best best desired outcome, but a nice workaround, yes! :-) Thumps up.

Your second suggestion to use the hirarchy/order of the filters did not work for me as the e-mail list i am working with has too many e-mails with similar subject / body that means i need a very fine-grained approach.

Example of approach that would work:

Filter x: Run when ANY of A, B, C or D are fulfilled: A) condition 1 AND condition 2 --> move to folder x B) condition 1 AND condition 3 --> move to folder x C) condition 1 AND condition 4 --> move to folder x D) condition 1 AND condition 5 AND condition 6 AND condition 7 --> move to folder x

In the current version this can be done by just creating A LOT of filters (one filter for A, one filter for B, one filter for C, one filter for D ... ), which would leave me with a dozens of filters for the same folder, which is not desired. I guess especially with the search filter option (nice! :D) it would be managable, but it would not be nice to work with xD. Myself, and i assume a lot of other people, would prefer to work with one filter per folder.

Now that i think about it, another workaround would be make the AND (and ANY) function an option in the submenu, allowing more than one AND function.

Sorry, if i have a lot of ideas. My programming skills are not very developed, but i know actually implementing them is not as easy as coming up with ideas 😅

[ThiloteE]

The initial suggestion in Bug 294632 on Bugzilla (16 (!) years ago) mostly captures my feelings towards this topic. The conversation is long and deals with various workaround, but seems a little bit outdated though. Last entry was 7 years ago. https://bugzilla.mozilla.org/show_bug.cgi?id=294632

Similar to Bug 294632. Mentions a workaround and got tagged "dublicate". https://bugzilla.mozilla.org/show_bug.cgi?id=933795

The following entry on bugzilla deals with the solution (folder as option within filter itself) you suggested: https://bugzilla.mozilla.org/show_bug.cgi?id=1603823

---

This entry follows up on Boolean Expressions in the filter routine (AND, OR, ...). https://bugzilla.mozilla.org/show_bug.cgi?id=297852 Apparently the filter engine is able to do it, but the UI is missing.

This one shows another suggestion how a filter menu could look like with working Bulean expressions but got tagged as dublicate as well. (Mine would look different! Needs more AND possibilities xD). https://bugzilla.mozilla.org/show_bug.cgi?id=378343

[ThiloteE]

I am about to write some comments on bugzilla. Shall i mention this issue over there? Would you mind if i post/link your suggestion of a possible solution in https://bugzilla.mozilla.org/show_bug.cgi?id=1603823?

[RealRaven2000]

I am about to write some comments on bugzilla. Shall i mention this issue over there? Would you mind if i post/link your suggestion of a possible solution in https://bugzilla.mozilla.org/show_bug.cgi?id=1603823?

Sure you can, that's why github is open source. Of course I cannot guarantee that I can pull off any low level solutions that work around Thundebird limitations in my Add-ons, what with their move towards only allowing what they expose in form of APIs for the future, anyway.

[RealRaven2000]

Maybe a more logical place to put the folder selection would be in the condition box:

One problem here is that you may want to have a mix of "any condition" for all other search conditions combined with always running on the folder, so it may be a little harder to make an edge case here. Best would be if the folder selector was beside the checkbox "Getting New Mail" as an "alternative choice":

• this solution is slightly ambiguous as it should also prevent it from running in the Inbox? Also it would require structural changes in the filter's architecture, whereas adding it as a search condition or action doesn't break / extend the existing model.

[RealRaven2000]

Related bug in FiltaQuilla:

https://github.com/RealRaven2000/FiltaQuilla/issues/88

Enhancement: Apply Filter automatically to messages outside Inbox

[ThiloteE]

How about both? 👯

I know it is tempting to put it into the filter after/before junk menu, but would this not

A) make it impossible to select to filter after/before junk? and B) it would be quite hidden. People would need to search for this feature.

So, if it is possible, why not go the whole hog? Sorry, wants are limitless i know xD

[ThiloteE]

The whole hog:

Having it in the condition box would be awesome too, but as you said, using it with match any of the following would be difficult.

[RealRaven2000]

How about both? dancers

I know it is tempting to put it into the filter after/before junk menu, but would this not

A) make it impossible to select to filter after/before junk?

• junk controls only run in the inbox, so that's irrelevant

and B) it would be quite hidden. People would need to search for this feature.

true. But it's also a highly non-standard feature that's actually not supported by Thunderbird and it may seriously affect performance if you have many filters that need to be interrogated - a folder listener would be triggered (possible multiple times) on every new email that arrives in the particular folder. But a bigger problem is that you could only select a single folder - so adding to search terms or conditions would definitely be superior from an UI perspective.

THe main problem here is that quickFilters would basically have to built its own external watchdog mechanism that would need to be instantiated on app start (read all filters, find the ones that act on other folders, create folder listeners with individual lists of which filters should run there, and run these filters, all in JavaScript - not using the inbuilt mechanism which uses C++) so it is quite a big ask, and potentially it's going to be buggy and slow, so it might a big wasted effort.

If we could convince the Thunderbird team to take this on on the lower filter level it would be way better!

[RealRaven2000]

The whole hog:

that takes a lot of screen estate (the dialog already "crams" the search conditions list very much. In my mind the top terms box should take up 2/3 of the height, and you can see in your example it's down to 1/5.

As I said in my previous comment, what if you want to run the filter on 2 other folders? So maybe in that case the actions box would actually be the best place (but also awkward to program). The main problem is establishing all the folders that have this set and then listening there and activating the correct filters. Basically reinventing the wheel...

[ThiloteE]

THe main problem here is that quickFilters would basically have to built its own external watchdog mechanism that would need to be instantiated on app start (read all filters, find the ones that act on other folders, create folder listeners with individual lists of which filters should run there, and run these filters, all in JavaScript - not using the inbuilt mechanism which uses C++) so it is quite a big ask, and potentially it's going to be buggy and slow, so it might a big wasted effort.

Would all the solutions you propose have to be done with javascript and do its own watchdog mechanism?

---

I think there are five different options on the table right now:

• [ ] o) as separate option in Apply filter when: menu.
• [ ] oplus) as separate option in Apply filter menu + new box that allows to select multiple folders.
• [ ] p) In dialogue Getting new mail beneath filter after junk classification (WITHIN dialogue).
• [x] q) in Perform these actions box

• [ ] r) in the conditions box

  Albeit awkward to use, all these five options are better than having none of them xD

  If i understand you right, you would currently not favour options o and oplus, because

• performance issues (multiple folder listeners)
• better done in c++
• A lot of thunderbird UI would need to be restructured

• takes of lot of UI space

  (although i think oplus is the best option for the user out of all of them)

  You would currently not favour option p, because

• would require structural changes in the filter's architecture
• one would not be able to select multiple folders to run on (without substantial change in UI)

Not favour option r, because

• issues with match any of the following
• issues if filter is supposed to run on multiple folders
• (although it also would not require architectural changes)

You would currently favour option q because

• does not need changes in filter architecture
• does not require new UI
• awkward to program, but doable

• allows match any and all of the following conditions to run on one or multiple folders

  Seems like we have a clear winner here:

  Option q seems to be it. At least for now. Until Thunderbird core developers decide to put filter issues higher on the agenda and attempt to do a rework.

[ThiloteE]

For future reference:

option oplus

[RealRaven2000]

THe main problem here is that quickFilters would basically have to built its own external watchdog mechanism that would need to be instantiated on app start (read all filters, find the ones that act on other folders, create folder listeners with individual lists of which filters should run there, and run these filters, all in JavaScript - not using the inbuilt mechanism which uses C++) so it is quite a big ask, and potentially it's going to be buggy and slow, so it might a big wasted effort.

Would all the solutions you propose have to be done with javascript and do its own watchdog mechanism?

Yes. I would have to implement via a "folder listener" in JavaScript. Add-ons cannot add binary / compiled code so C++ is not an option.

I think there are five different options on the table right now:

• [ ] o) as separate option in Apply filter when: menu.
• [ ] oplus) as separate option in Apply filter menu + new box that allows to select multiple folders.
• [ ] p) In dialogue Getting new mail beneath filter after junk classification (WITHIN dialogue).
• [x] q) in Perform these actions box
• [ ] r) in the conditions box

Albeit awkward to use, all these five options are better than having none of them xD

If i understand you right, you would currently not favour options o and oplus, because

• performance issues (multiple folder listeners)
• better done in c++

all of the options have these problems. no matter how you present this in the UI. Behind the scenes you will have to read every filter, check whether it needs automatical running in other folders, create folder listeners that react to new mail and run the filter automatically. Changing the UI does not change the requirements on what the backend does.

• A lot of thunderbird UI would need to be restructured
• takes of lot of UI space

If we take the "comfortable route" with multiple lines of folder selectors then it wont all fit into this dialog, the UI is already hopelessly complex. So a button opening a completely new dialog where different folders can be selected into a list would be the solution for this.

(although i think oplus is the best option for the user out of all of them)

You would currently not favour option p, because

• would require structural changes in the filter's architecture

That's another problem - but applies to most of these solutions: quickFilters uses the current architecture and cannot extend it. Because the filters are stored by the back-end in a way that no Add-on can change. That's why it would be easier to stuff folder locations into one of the lists (search terms or actions). However it would probably at least be a little easier to add another choice to the "Getting new mail" dropdown, as that is just stashed in a flag IIRC.

• one would not be able to select multiple folders to run on (without substantial change in UI)

One could have a button [select folders..] besides that becomes visible / enabled after selecting the option "In Other Folders"

Not favour option r, because

• issues with match any of the following
• issues if filter is supposed to run on multiple folders

well it could, but only as long as you select "ANY of these conditions". One might argue that the logic should handle the "AND" case by silently inserting parentheses:

x AND y AND (folder1 OR folder2 OR folder3)

internally this is already supported. It's just almost impossible to build a UI for that.

• (although it also would not require architectural changes)

You would currently favour option q because

• does not need changes in filter architecture
• does not require new UI
• allows match any and all of the following conditions to run on one or multiple folders
• awkward to program, but doable

yeah but hopelessly complex. It would have to manage folder listeners on App start (first problem is to find out when filters have finished loading, not sure whether there is an event for that) and then again whenever users add / remove / change filters - existing folder listeners would have to be deleted / recreated. Also "allows match any and all of the following conditions" that's not true, it would strictly be "any" for the folders and independent for the other search conditions (which cannot be mixed).

[ThiloteE]

well it could, but only as long as you select "ANY of these conditions". One might argue that the logic should handle the "AND" case by silently inserting parentheses:

x AND y AND (folder1 OR folder2 OR folder3)

internally this is already supported. It's just almost impossible to build a UI for that.

in regards to UI for boolean logic, how about:

as a quick low hanging fruit?

I name this proposal AO.

[ThiloteE]

So ... with regard to the filters outside of inbox issue, which solution would you ultimately prefer?

[RealRaven2000]

well it could, but only as long as you select "ANY of these conditions". One might argue that the logic should handle the "AND" case by silently inserting parentheses: x AND y AND (folder1 OR folder2 OR folder3) internally this is already supported. It's just almost impossible to build a UI for that.

in regards to UI for boolean logic, how about:

as a quick low hanging fruit?

No it isn't - this is absolutely NOT low hanging fruit. There is an extensive discussion here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1465076

and a feature request on my Add-on FiltaQuilla here: https://github.com/RealRaven2000/FiltaQuilla/issues/113

...and your screenshot make no sense - it can't be done that way - we can't just add new lines to control operands in search conditions.

Every condition internally has 2 flags, beginsGrouping and endsGrouping, (opening and closing parenthesis) and its operator (and BooleanAND flag). The core team would have to implement UI to make these things visible in way that a user might understand them:

https://udn.realityripple.com/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIMsgSearchTerm https://udn.realityripple.com/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIMsgFilter

btw. the documentation for these interfaces was removed from the official mozilla developer network over 3 years ago as we (Add-on developers) are discouraged from using these, see:

[ThiloteE]

Ah yes. Foolish of me to think it could be that easy.

[RealRaven2000]

Ah yes. Foolish of me to think it could be that easy.

There are several confounding things for the complexity:

• the UI doesn't really say anything on how the backend is organized. Always assume it's totally dumbed down and paints over a lot of cracks.
• building UI is hard. Especially one that does complex things (like boolean logic) and can be understood by the layman
• there are at least 2 different layers: the javascript code for the front end and C++ code (compiled) for the backend. In the filtering mechanism these are really badly mixed up
• a lot of the platform code (the app + browser) is handed down from Firefox who don't have any mail specific code see mozilla-central. The Thunderbird team then adds its own code base on top - see comm-central. Most of the (web extension) Add-on architecture plus the related public facing websites are owned by Mozilla (the Firefox team). Thunderbird has very few core developers but they have people (or maybe just 1 person) working on mail specific APIs. The backend code is not documented but can be publicly viewed at searchfox.org.
• this causes everything that relies on non-documented functions to break whenever a new version gets released by Firefox (every 4 weeks) - Thunderbird avoids most of these breakages by only merging in security relevant changes and building a parallel version that gets released once a year (ESR). That's when "experimental" Add-on authors are usually panicking because a lot of their stuff stops working.
• there currently is no filtering API. The current API documentation is here. This and the searchfox results in the main code base is basically our main source for information on how to do stuff...

[ThiloteE]

Thank you for the explanation. That helped a little bit at reducing complexity 😅

[ThiloteE]

Ugly workaround for running automatically outside inbox:

Creaty "dummy account" that acts as intermediate inbox folder for specific filter rules.

This dummy account could be a:

1. inbox of new e-mail account
   2. inbox of new local folder
   3. inbox of new feed (= roughly the same as a local folder)

Workflow:

1. Receive mail in inbox 1
2. Run filters (X) on inbox 1
   • Filters (X) move or copy mails from inbox 1 to Dummy Inbox
3. Automatically and periodically run filters (Y) on Dummy inbox