Header Regex, "Received" no longer works?

[mabar66]

I can't filter the content of the headers:

Works: Header Regex Match => matches => Received:

Does not work : Header Regex Match => corresponds to =>Received: from cgrd-ne.com

??????

[RealRaven2000]

can you make a screenshot of the filter or export it using the quickFilters export button (you only have that if you install the quickFilters Add-on):

[mabar66]

it's Filtaquilla addon thunderbird in linux !

[RealRaven2000]

did you make it work? If yuou can export the email that matches as "eml" format I can test. I guess you have to remove the "from" and maybe escape the "." - like this:

Received:cgrd-ne\.com

[mabar66]

thanks ...i test !

[mabar66]

I can't attach the files??

​it does not work !

[RealRaven2000]

I can't attach the files??

​it does not work !

If .eml files are not allowed just archive them in a zip file. (HTML or similar files may not be allowed here on Github as attachments) zip files always work. btw I will be out of office tomorrow, hopefully have time to look at this Friday again.

[mabar66]

thanks mail.zip

[mabar66]

thanks mail.zip

[mabar66]

can you look at my zip?

[mabar66]

I tested the regex on the regex101.com site (ECMAScript) and it works.... Received: from cgrd-ne.com ????

[mabar66]

i test : Received: /from cgrd-ne.com/gm Received: /from cgrd-ne.com/ ????

[mabar66]

Can you confirm that your addon works for me? linux mint 21.2 thunderbird 115.5.0 THANKS

[mabar66]

I found the answer to my problem... With your addon it doesn't work!!!!

but you can add search words to the Thunderbird filter list in Firefox!

To add simple headers to the search list in the source file:

under Firefox in the address bar type: about:config search: mailnews.customDBHeaders Add your header: Received

then go to the filter, it should appear in the list.....

i test : Received=> contient =>cgrd-ne.com

tested....works!!!

[RealRaven2000]

I am investigating, apparently the command aMsgHdr.getStringProperty("Received"); returns an empty string. This part of the code is the original code from Kent James, hopefully there is another way to get at the data from the JavaScript layer - bear in mind this i all very low level, none of it is supported by the mail extension API and nothing is documented, so I have to dig around in the Thunderbird Core code over at searchfox.org to see how the heck they display it in the header area.

i will start by finding out which code populates the area above the email preview..

[RealRaven2000]

Apparently the mail header area fills the object currentHeaderData (a temporary object that is onbly accessible when the email is selected, so it's not existent when filters are run in any folder) during an event function onHeadersComplete(mailChannel) - I haven't found how this event function is called, but I suspect it may happen from the C++ / backend code...

https://searchfox.org/comm-esr115/source/mail/base/content/msgHdrView.js#568

I need to know where and how the mailChannel variable is filled with data...

[RealRaven2000]

As I suspected. it's called from the C++ back end code (something I cannot look at from the JavaScript only Add-on debugger):

I need to ask a Thunderbird Core developer on the next Add-on developers call whether it is possible to harness this function at all from JavaScript. (They encourage us to use API functions instead, but there is absolutely 0 stuff there for filters. The filter mechanism is run by the back end code from the compiled area of the software we barely have enough power to register new search methods. Supporting low level access to header was never planned by the Thunderbird development team. I can try and ask anyway, otherwise I can only retire the function.

[RealRaven2000]

I found the answer to my problem... With your addon it doesn't work!!!!

but you can add search words to the Thunderbird filter list in Firefox!

To add simple headers to the search list in the source file:

under Firefox in the address bar type: about:config search: mailnews.customDBHeaders Add your header: Received

Firefox? You can open the config databas in Thunderbird general settings. scroll down to the button on the bottom right [Config Editor...]

However to add a custom header here the official way is to open filter editor and then select "Customize..." on the search term dropdown:

Then add received as new message header:

I tried creating the filter like you described, with received contains cgrd-ne.com:

• but it didn't tag the message you sent for me.

[mabar66]

This is how I did it and it doesn't work !!!

Apply the one I indicated which works... =>

Under Firefox in the address bar type: about:config search: mailnews.customDBHeaders Add your header: Received

then go to the filter, it should appear in the list.....

[RealRaven2000]

Firefox. I thought we are doing Thunderbird here - how do you make a filter in Thunderbird that reads settings from Firefox?

[RealRaven2000]

(The configuration database can be edited in Thunderbird as well - no need to open Firefox) Thunderbird Settiongs / General / scroll down and click [Config Editor]

It doesn't make sense to me that you are using filters in thunderbird if your settings are in Firefox (which stores its configuration in a completely different space)

[RealRaven2000]

Here is a test version with more information in the console - this will list all headers that are currently available to the msgHeader object, if you enable a new special debug option.

filtaquilla-4.0pre40.zip

to activate the debug log for header regex, open FiltaQuilla options and then open the About FQ/Help page. Activate debug Mode (bottom right) right-click debug mode and then activate extensions.filtaquilla.debug.regexHeader = true

---

to install the test version, download the zip file and then drag the file into Thunderbird Add-ons Manager (do not extract contents).

[RealRaven2000]

This is the list of readable string properties from your email. We have Add-on developer meeting this evening and I will try to find out why the Received headers are not available in Thunderbird during filtering anymore, or what alternative method can be used to retrieve these headers. I suspect I will have to go back to the message database and read the complete content to make it work...

flags, sender, recipients, subject, message-id, references, dateReceived, date, msgCharSet, sender_name, 
offlineMsgSize, msgOffset, storeToken, priority, size, threadParent, msgThreadId, ProtoThreadFlags, 
X-GM-MSGID, X-GM-THRID, X-GM-LABELS, remoteContentPolicy, gloda-id, keywords

[RealRaven2000]

The compiled function of the filter service is here nsMsgFilterList::ApplyFiltersToHdr()

one of the parameters is this (presumabily reference of string or strings) const nsACString& headers

this happens while iterating through the active filters:

filter->MatchHdr(msgHdr, folder, db, headers, &result);

source is here: https://searchfox.org/comm-esr115/source/mailnews/search/src/nsMsgFilter.cpp#563

This calls

 nsMsgSearchOfflineMail::MatchTermsForFilter(
      msgHdr, m_termList, folderCharset.get(), m_scope, db, headers,
      &m_expressionTree, pResult);

I think this is implemented in this C++ file nsMsgLocalSearch.cpp:

nsresult nsMsgSearchOfflineMail::MatchTermsForFilter(
    nsIMsgDBHdr* msgToMatch, nsTArray<RefPtr<nsIMsgSearchTerm>> const& termList,
    const char* defaultCharset, nsIMsgSearchScopeTerm* scope,
    nsIMsgDatabase* db, const nsACString& headers,
    nsMsgSearchBoolExpression** aExpressionTree, bool* pResult) {
  return MatchTerms(msgToMatch, termList, defaultCharset, scope, db, headers,
                    true, aExpressionTree, pResult);
}

MatchTerms is defined in nsMsgLocalSearch.cpp :

nsresult nsMsgSearchOfflineMail::MatchTerms(
    nsIMsgDBHdr* msgToMatch, nsTArray<RefPtr<nsIMsgSearchTerm>> const& termList,
    const char* defaultCharset, nsIMsgSearchScopeTerm* scope,
    nsIMsgDatabase* db, const nsACString& headers, bool Filtering,
    nsMsgSearchBoolExpression** aExpressionTree, bool* pResult) {
  NS_ENSURE_ARG(aExpressionTree);
  nsresult err;

  if (!*aExpressionTree) {
    uint32_t initialPos = 0;
    uint32_t count = termList.Length();
    err = ConstructExpressionTree(termList, count, initialPos, aExpressionTree);
    if (NS_FAILED(err)) return err;
  }

  // evaluate the expression tree and return the result
  *pResult = (*aExpressionTree)
                 ? (*aExpressionTree)
                       ->OfflineEvaluate(msgToMatch, defaultCharset, scope, db,
                                         headers, Filtering)
                 : true;  // vacuously true...

  return NS_OK;
}

note: this is all c++ code which is not directly accessible to Add-ons and cannot be overwritten even via monkey-patching because it is in the compiled version of thunderbird.exe (it is not directly scriptable).

[RealRaven2000]

...however it would be possible to debug this with a binary debugger, it's just slow and complicated. the end goal would be to find out why 115 does not expose the received (and other) headers to the nsIMsgDBHdr interface:

https://searchfox.org/comm-esr115/source/mailnews/base/public/nsIMsgHdr.idl#14

comparing the latest comm-central release with that of Thunderbird 91 isn't significantly different, structure-wise. So my guess is that some parts may not be filled with the same amount / type of data anymore; more investigation is needed.

[RealRaven2000]

Yesterday, we had the first Add-on developers meeting in the new year. John Bieling who is Thunderbird Employee told me about the following Mozilla bug, that deals with the received header not being exposed to the message header since Thunderbird 102.4.2:

Bug 1800759

There is a fix in the pipeline, so my hope is that it may be exposed again. However Magnus Melin said that some headers may not be available to the programming interfaces deliberately.

Another option to access the raw headers would be by using the new web extension APIs - but the problem here is that their methods are mostly asynchronous and thus they cannot be called from a filter condition - as long as all filter functions in Thunderbird are designed as synchronous functions. My hope is that eventually search terms and filter actions will be rewritten asynchronously which will allow us to use a lot more of the modern functionality in Thunderbird. Many old functions in Tb (especially those dealing with files and web URIs) have already been converted into asynchronous methods, but this is a big piece of work that will need a dedicated, experienced staff member to adopt it and get funding / priority.

I will continue to lobby getting filters async in the future until it is hopefully implemented.

[xlovinglyx]

i get it.. headers regex's dead since 102.4.2 while filtaquilla keeps being updated as if nothing happened.

[RealRaven2000]

i get it.. headers regex's dead since 102.4.2 while filtaquilla keeps being updated as if nothing happened.

any suggestions on how it could be fixed??