Support FiltaQuilla search terms on News/Usenet

[bilo1967]

Maybe I'm missing something but I can't find most of FiltaQuilla filters for news/usenet posts. There are for email accounts but not for News accounts. In particular, I can't find Subject Regexp Match. There's the Javascript filter (and JavascriptAction) and I can use it instead of it like this:

message.getStringProperty('subject').match(/something/i);


and of course this way it's possible to build much more complex conditions, but they are harder to read and most of the times a simple regexp would be enough.

May be I'm doing something wrong? Or is there a reason why the Subject Regexp Match isn't implemented?

[RealRaven2000]

I don't know I have to test it out... can you recommend a particular newsgroup for testing?

[bilo1967]

No, no newsgroup in particular. Maybe it.test, which is for test and has enough posts (it's full of crap).

You can use news.neodome.net or freenews.netfront.net as a server (readonly) or news.eternal-september.org (both reading and posting).

Thank you for giving a look!

P.S. The "symptom" is simply that FiltaQuilla filters do not appear. There's only Javascript (and it works).

[RealRaven2000]

or news.eternal-september.org (both reading and posting).

now there's a trip down memory lane, I haven't used that server in a looong long time. I hope I can think of it will ask the question whether there is something special about newsgroups that might make filters fail on tomorrow evening's Add-on Developers meeting.

[RealRaven2000]

Also I will run some tests with a debugger. maybe I can see some warnings / errors in browser console (Ctrl+Shift+J)

[RealRaven2000]

I see. it may be a design restriction of Thunderbird itself, but to me it looks like an oversight. I hope John Bieling can give me some insight in tonight's Add-on Developer meeting...

[bilo1967]

Yes, that's also what I see. P.S. The problem with the Javascript approach is that the "subject" property is raw, while some subjects are mime encoded. And as far as I can see the "message" nsIMsgDBHdr object here does not provide a "mime2DecodedSubject" property.

[RealRaven2000]

And as far as I can see the "message" nsIMsgDBHdr object here does not provide a mime2DecodedSubject property.

where did you check, debugger?

This interface should still exist at least in Tb 115: https://searchfox.org/comm-esr115/source/mailnews/base/public/nsIMsgHdr.idl#87

Also in daily (Comm-central): https://searchfox.org/comm-central/source/mailnews/base/public/nsIMsgHdr.idl#87

[bilo1967]

I don't know anything about TB API and I did a simple brute-force approach. I posted a message with a subject containint XXAAXX on it.test and setup a javascript filter with the following code:

if ( message.getStringProperty('subject').match(/XXAAXX/i) ) {
    Cc["@mozilla.org/consoleservice;1"]
         .getService(Ci.nsIConsoleService)
         .logStringMessage('properties: ' + message.properties);
    1;
} 

On the debugger window I get this:

properties: flags,subject,sender,date,message-id,references,size,numLines,threadParent,msgThreadId,ProtoThreadFlags,sender_name,keywords

I've also tried directly with either

message.getStringProperty('mime2DecodedSubject').match(/XXAAXX/i);

and

message.getStringProperty('mime2decodedsubject').match(/XXAAXX/i);

but they didn't work. Maybe I'm trying to access properties in the wrong way? Let me know if I can help you somehow.

[bilo1967]

Ok, it actually works in the simplest way:

message.mime2DecodedSubject.match(/any pattern/i);

[RealRaven2000]

yes it's a direct attribute, I hope John may help finding an answer about newsgroups. AS Add-on devs we are asked to use the official APIs but with filters it's not really possible as there is no API for filter actions / and searching. I am still using "experimental" code here and it is largely based on the undocumented front end code that can be found over at searchfox.org. Most of it I learned by debugging the code, but filters are very very deep below the surface because the filtering service is written in C++ and only dips into JS for the custom actions / search terms. And I can't really debug the C++ back-end it's too much work for me.

[bilo1967]

Don't know it this can help. I tried to force the "Subject Regex Match" filter by manually adding a rule to the existing msgFilterRules.dat file in a newsgroup account directory, like this:

version="9"
logging="yes"
name="test 2"
enabled="yes"
type="20"
action="Mark read"
condition="AND (filtaquilla@mesquilla.com#subjectRegex,matches,XXAAXX)"

When I open the rules panel, go to "Modify" and press "Ok" I get a message (in italian) which sounds like "This filter cannot be saved because the search of "Subject Regex Match is equal" isn't valid in current context.".

The filter name is actually resolved but for some reason cannot be used for newsgroups.

[RealRaven2000]

We found the problem yesterday - it's a function _isLocalSearch that only allowed certain search scopes

  function _isLocalSearch(aSearchScope) {
    switch (aSearchScope) {
      case Ci.nsMsgSearchScope.offlineMail:
      case Ci.nsMsgSearchScope.offlineMailFilter:
      case Ci.nsMsgSearchScope.onlineMailFilter:
      case Ci.nsMsgSearchScope.localNews:
        return true;
      default:
        console.log("isLocalSearch == FALSE!", aSearchScope);  // test!!!
        return false; // test!!! 
    }
  }

I added Ci.nsMsgSearchScope.newsFilter for now - but there may be some cases where it may makes sense not to allow the search (maybe body regex might fail, not sure)

filtaquilla-4.0pre45.zip

---

To test the version above, download the zip file, drag the file into Thunderbird Add-ons Manager, do not extract contents or if won't install.

[bilo1967]

We found the problem yesterday - it's a function _isLocalSearch that only allowed certain search scopes

Cool! Thank you so much!

I added Ci.nsMsgSearchScope.newsFilter for now - but there may be some cases where it may makes sense not to allow the search (maybe body regex might fail, not sure)

As far as I know a body regex would certainly fail during the fetch of new messages, since only headers are fetched and bodies are retrieved when clicking the message. I've also tried to manually execute the filter after downloading some messages but it still does not work.

In the log there's this error.

Uncaught NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIMsgFolder.getMsgInputStream]
    bodyMimeMatch chrome://filtaquilla/content/filtaquilla-util.js:518
    bodyRegEx_match chrome://filtaquilla/content/filtaquilla.js:1670
    _runFilters resource:///modules/NntpNewsGroup.jsm:319
    finishProcessingXOver resource:///modules/NntpNewsGroup.jsm:202
    _actionXHdr resource:///modules/NntpClient.jsm:707
    _actionReadXOver resource:///modules/NntpClient.jsm:691
    read resource:///modules/LineReader.jsm:65
    _actionReadXOver resource:///modules/NntpClient.jsm:683
    _onData resource:///modules/NntpClient.jsm:219
    editFolder chrome://messenger/content/about3Pane.js:3196
    <anonymous> chrome://messenger/content/about3Pane.js:6602
    doCommand chrome://messenger/content/mailCommon.js:729
    onCommand chrome://messenger/content/about3Pane.js:585
    handleEvent chrome://messenger/content/about3Pane.js:285
249 filtaquilla-util.js:518
    bodyMimeMatch chrome://filtaquilla/content/filtaquilla-util.js:518
    bodyRegEx_match chrome://filtaquilla/content/filtaquilla.js:1670
    _runFilters resource:///modules/NntpNewsGroup.jsm:319
    finishProcessingXOver resource:///modules/NntpNewsGroup.jsm:202
    _actionXHdr resource:///modules/NntpClient.jsm:707
    _actionReadXOver resource:///modules/NntpClient.jsm:691
    read resource:///modules/LineReader.jsm:65
    _actionReadXOver resource:///modules/NntpClient.jsm:683
    _onData resource:///modules/NntpClient.jsm:219
    editFolder chrome://messenger/content/about3Pane.js:3196
    <anonima> chrome://messenger/content/about3Pane.js:6602
    doCommand chrome://messenger/content/mailCommon.js:729
    onCommand chrome://messenger/content/about3Pane.js:585
    handleEvent chrome://messenger/content/about3Pane.js:285

It certainly appears when fetching headers. I have not checked if also there when manually executing the rule.

To test the version above, download the zip file, drag the file into Thunderbird Add-ons Manager, do not extract contents or if won't install.

Did som testing right now. Works with header regexp match and subject regexp match. I'll keep testing.

[RealRaven2000]

As far as I know a body regex would certainly fail during the fetch of new messages, since only headers are fetched and bodies are retrieved when clicking the message. I've also tried to manually execute the filter after downloading some messages but it still does not work.

In the log there's this error.

thanks for testing this, I think I can just disable it for body regex in this case to be safe. Thunderbird vanilla doesn't offer Body here either.

[bilo1967]

I think I can just disable it for body regex in this case to be safe. Thunderbird vanilla doesn't offer Body here either.

yes. definitely the right solution.