static filter rules

[pazz]

Hi, just tried to build custom filters to reflect my static filtering rules. I presume the syntax below is as intended?

# global configuration
[global]

# This is the default filter chain
[SpamFilter]
#[ClassifyingFilter]
#[KillThreadsFilter]
[ListMailsFilter]
[ArchiveSentMailsFilter]

[Filter.1]
query = peergroup
tag = +peergroup
message = ****peergroup filter*

[Filter.2]
query = subject:alot
tag = +alot
message = ****alot*****

[InboxFilter]

i have a message tagged with new matching the query "is:new AND peergroup", but afew wouldn't tag it according to the log:

pazz@brick:~/projects/afew$ afew -t --new -d -vv
INFO:root:Tagging spam messages
DEBUG:root:Executing query u'(tag:new)'
INFO:root:I would commit changes to 0 messages
INFO:root:Tagging mailing list posts
DEBUG:root:Executing query u'((tag:new)) AND (NOT tag:lists)'
DEBUG:root:Adding tags lists, alot to Justus Winter <reply+i-2624298-4d926afc59836ca6477a8b3178d092598408302e-628455@reply.github.com> (inbox new) (2011-12-21)
DEBUG:root:Adding tags lists, peergroup to Daniel Kreischer <d.t.kreischer@gmx.de> (inbox killed list new) (2011-12-20)
DEBUG:root:Adding tags lists, peergroup to Sascha Steinbiss <satta@tetrinetsucht.de> (inbox killed list new) (2011-12-19)
DEBUG:root:Adding tags lists, peergroup to Daniel Kreischer <d.t.kreischer@gmx.de> (inbox killed list new) (2011-12-19)
INFO:root:I would commit changes to 4 messages
INFO:root:Archiving all mails sent by myself
DEBUG:root:Executing query u'((tag:new)) AND (from:"patricktotzke@gmail.com" OR from:"p.totzke@ed.ac.uk")'
INFO:root:I would commit changes to 0 messages
INFO:root:****peergroup filter*
DEBUG:root:Executing query u'((tag:new)) AND (peergroup)'
INFO:root:I would commit changes to 0 messages
INFO:root:****alot*****
DEBUG:root:Executing query u'((tag:new)) AND (subject:alot)'
INFO:root:I would commit changes to 0 messages

[teythoon]

just tried to build custom filters to reflect my static filtering rules. I presume the syntax below is as intended?

Well, I use a more explicit queries in my custom filters:

[Filter.1]
query = subject:"[Mafia]"
tags = +studium

But then again, there is no reason why peergroup shouldn't be a valid query if notmuch search -- peergroup returns a valid result. I'll look into it.

[teythoon]

But then again, there is no reason why peergroup shouldn't be a valid query if notmuch search -- peergroup returns a valid result. I'll look into it.

Hm, strange. I cannot reproduce this. I set up a filter with a query similar to yours:

[Filter.2]
query = test
tags = +test

And sent me a mail:

teythoon@thinkbox ~/repos/afew (git)-[master] % notmuch search -- tag:new and test
thread:0000000000005e4b  1 mins. ago [1/1] Justus Winter; test (attachment new unread)
thread:0000000000005e4c  1 mins. ago [1/1] Justus Winter; test (attachment new unread)
teythoon@thinkbox ~/repos/afew (git)-[master] % python3.2 bin/afew -vv --tag --new
[...]
DEBUG:root:Executing query '((tag:new)) AND (test)'
DEBUG:root:Adding tags test to Justus Winter <4winter@informatik.uni-hamburg.de> (alot attachment new unread) (2011-12-24)
DEBUG:root:Adding tags test to Justus Winter <4winter@informatik.uni-hamburg.de> (alot attachment new unread) (2011-12-24)
INFO:root:Committing changes to 2 messages
[...]
teythoon@thinkbox ~/repos/afew (git)-[master] % notmuch search -- tag:inbox and not tag:killed and tag:test
thread:0000000000005e4b  1 mins. ago [1/1] Justus Winter; test (alot attachment inbox test unread)
thread:0000000000005e4c  1 mins. ago [1/1] Justus Winter; test (alot attachment inbox test unread)

[pazz]

is it possible that the position of that definition in the config is relevant?

offtopic: i think the query string for the InboxFilter is a bit redundant:

INFO:root:Retags all messages not tagged as junk or killed as inbox
DEBUG:root:Executing query u'((tag:new)) AND ((tag:new))'

[teythoon]

Quoting Patrick Totzke (2011-12-24 13:08:32)

is it possible that the position of that definition in the config is relevant?

Well, sure, but as long as the InboxFilter is the last one (as it strips the new tag from all messages) it should be fine.

offtopic: i think the query string for the InboxFilter is a bit redundant:

INFO:root:Retags all messages not tagged as junk or killed as inbox
DEBUG:root:Executing query u'((tag:new)) AND ((tag:new))'

Indeed. I trust xapian to handle this gracefully and efficiently.

Justus love u alot @,@

[kazuoteramoto]

Hit this issue today. tags != tag @pazz filter use the singular, but the correct is the plural form.

[pazz]

uh.. good one! thanks kazuoteramoto! I already set up afew successfully by C&Psting a bit, but i'm glad this is settled then. btw: @teythoon: did you notice my afew-setup highlight in the channel? do you prefer issues here?

[teythoon]

@kazuoteramoto good catch indeed, thanks!

@pazz yes, I'm not that good with synchronous communication methods, so I usually prefer tickets here ;). I assume that this issue can be closed now? If not, please reopen this ticket.

[pazz]

i have another issue with these static filters. i'll reopen this here fore its appropriate topic:

i have this static filter in my chain, after the list filter:

[Filter.2]
query = 'to:notmuch@notmuchmail.org and subject:emacs'
tags = -inbox
message = notmuch emacs stuff

and i often get mails in my inbox that are tagged with notmuch, and lists. why does this happen? The filter seems to work:

$ afew --tag --dry-run is:inbox -v
INFO:root:Tagging spam messages
INFO:root:I would commit changes to 0 messages
INFO:root:Looking for messages in killed threads that are not yet killed
INFO:root:I would commit changes to 0 messages
INFO:root:Tagging mailing list posts
INFO:root:I would commit changes to 0 messages
...
INFO:root:notmuch emacs stuff
INFO:root:I would commit changes to 6 messages
...

i do this via cron:

/home/pazz/.local/bin/offlineimap -c /home/pazz/.offlineimaprc -o && /usr/local/bin/notmuch new && /home/pazz/.local/bin/afew --tag --new

any ideas?

edit: this is totally weird:

pazz@brick:~/bin$ notmuch tag +new -- is:inbox; afew --tag --new -v; notmuch count to:notmuch@notmuchmail.org AND subject:emacs AND is:inbox
INFO:root:Tagging spam messages
INFO:root:Committing changes to 0 messages
INFO:root:Looking for messages in killed threads that are not yet killed
INFO:root:Committing changes to 0 messages
INFO:root:Tagging mailing list posts
INFO:root:Committing changes to 0 messages
INFO:root:alot mails
INFO:root:Committing changes to 0 messages
INFO:root:notmuch emacs stuff
INFO:root:Committing changes to 6 messages
INFO:root:sicsa
INFO:root:Committing changes to 0 messages
INFO:root:boring seminars
INFO:root:Committing changes to 0 messages
INFO:root:superboring seminars
INFO:root:Committing changes to 0 messages
INFO:root:process algebra list
INFO:root:Committing changes to 0 messages
INFO:root:call for papers
INFO:root:Committing changes to 0 messages
INFO:root:foosoc
INFO:root:Committing changes to 0 messages
INFO:root:gmail spam
INFO:root:Committing changes to 0 messages
INFO:root:warsaw connection
INFO:root:Committing changes to 0 messages
INFO:root:Retags all messages not tagged as junk or killed as inbox
INFO:root:Committing changes to 37 messages
6

the last int is the msg count. this sould be 0.

[pazz]

could it be that the last filter [InboxFilter] accidentally gets the whole unfiltered bunch of new messages ? do i have to remove tag new in all static filters?

[pazz]

sry, no permissions to reopen...

[teythoon]

could it be that the last filter [InboxFilter] accidentally gets the whole unfiltered bunch of new messages?

Yes, if you want to stop the InboxFilter from adding the inbox tag you need to remove the new tag from messages. This isn't an accident though ;)

do i have to remove tag new in all static filters?

I use static filter rules to add tags, I still want them to end up in my inbox view.

Btw, currently the --dry-run switch does not produce exactly the same result as running afew without it. For that to work properly I need to emulate the addition and removal of tags better.

[pazz]

ah ok! so i don't really need to remove inbox as at that point (in the static filter and before the InboxFilter) this tag isn't present. All i need to do is remove new to stop messages from ending up in my inbox right? What if I want to do something crazy like say, let one filter cause a msg to skip the inbox and a later filter should still process that message?

[teythoon]

ah ok! so i don't really need to remove inbox as at that point (in the static filter and before the InboxFilter) this tag isn't present. All i need to do is remove new to stop messages from ending up in my inbox right?

yes.

What if I want to do something crazy like say, let one filter cause a msg to skip the inbox and a later filter should still process that message?

hm, I haven't thought about that. Do you have a valid use case or is this a theoretical thought?

[pazz]

What if I want to do something crazy like say, let one filter cause a msg to skip the inbox and a later filter should still process that message?

hm, I haven't thought about that. Do you have a valid use case or is this a theoretical thought? I don't have a use case for this, just wanted to ask if it'd be possible. This is really unrealistic though because for this to be a problem you'd have to have a non serializable filter logic.. never mind :) thx, /p

[pazz]

I am still having peculiarities going on here:

this is the updated relevant part of my config:

[global]

# This is the default filter chain
[SpamFilter]
[KillThreadsFilter]
[ListMailsFilter]
...
[Filter.2]
query = 'to:notmuch@notmuchmail.org and subject:emacs'
tags = -inbox -new
message = notmuch emacs stuff
...
[InboxFilter]

I just received an email with subject [PATCH 2/2] emacs: Add more processing of .. which ended up in my inbox after (cron ran) notmuch new and afew filters.

if i try to filter the inbox manually afterwards, the filter apparently hits that mail, logs that it would tag it, but doesnt:

pazz@megatron:~$ afew --tag is:inbox -v
INFO:root:Tagging spam messages
INFO:root:Committing changes to 0 messages
INFO:root:Looking for messages in killed threads 
INFO:root:Committing changes to 0 messages
INFO:root:Tagging mailing list posts
INFO:root:Committing changes to 0 messages
INFO:root:alot mails
INFO:root:Committing changes to 1 messages
INFO:root:notmuch emacs stuff
INFO:root:Committing changes to 0 messages
INFO:root:sicsa
INFO:root:Committing changes to 0 messages
INFO:root:boring seminars
INFO:root:Committing changes to 0 messages
INFO:root:superboring seminars
INFO:root:Committing changes to 0 messages
INFO:root:process algebra list
INFO:root:Committing changes to 8 messages
INFO:root:call for papers
INFO:root:Committing changes to 0 messages
INFO:root:foosoc
INFO:root:Committing changes to 0 messages
INFO:root:gmail spam
INFO:root:Committing changes to 0 messages
INFO:root:warsaw connection
INFO:root:Committing changes to 0 messages
INFO:root:Retags all messages not tagged as junk 
INFO:root:Committing changes to 0 messages

[pazz]

Update: sorry, the filter wasn't right:

pazz@megatron:~$ notmuch search 'is:inbox AND subject:emacs'
pazz@megatron:~$ notmuch search 'is:inbox AND subject: emacs'
thread:0000000000002b17  Today 07:19 [1/18] David Edmondson| Dmitry Kurochkin; [PATCH 3/4 v3] test: Add test for Original Message hiding at point-min. (inbox lists notmuch signed unread)

also i misinterpreted the log: the filter before the emacs one hit something. Nevertheless: if i tag is:inbox nothing actually gets written:

pazz@megatron:~$ afew --tag is:inbox -v
INFO:root:Tagging spam messages
INFO:root:Committing changes to 0 messages
INFO:root:Looking for messages in killed threads that are not yet killed
INFO:root:Committing changes to 0 messages
INFO:root:Tagging mailing list posts
INFO:root:Committing changes to 0 messages
INFO:root:alot mails
INFO:root:Committing changes to 1 messages
INFO:root:notmuch emacs stuff
INFO:root:Committing changes to 1 messages
INFO:root:sicsa
INFO:root:Committing changes to 0 messages
INFO:root:boring seminars
INFO:root:Committing changes to 0 messages
INFO:root:superboring seminars
INFO:root:Committing changes to 0 messages
INFO:root:process algebra list
INFO:root:Committing changes to 8 messages
INFO:root:call for papers
INFO:root:Committing changes to 0 messages
INFO:root:foosoc
INFO:root:Committing changes to 0 messages
INFO:root:gmail spam
INFO:root:Committing changes to 0 messages
INFO:root:warsaw connection
INFO:root:Committing changes to 0 messages
INFO:root:Retags all messages not tagged as junk or killed as inbox
INFO:root:Committing changes to 0 messages

no changes are committed afterwards

[teythoon]

no changes are committed afterwards

well, isn't that exactly what you wanted, to stop the catchall at the end of the filter chain to add the inbox tag to the mail in question?

[pazz]

in that coase not: note the parameter 'is:inbox' : i tell him to filter all those messages in the inbox. The emacs filter finds one and tells me it commits '-inbox -new'. so afterwards i expect those message out of my inbox.

nevertheless the problem is of course that they even got there in the first place. after all, if the emacs filter finds them now, why not beforehand, removed the new and inbox tags, and made them skip the catchall filter at the end of the chain.. or am i missing something here?

[pazz]

i think i missed that people might change the subject of the topic! cf 87vcny7i2z.fsf@zancas.localnet. I guess this new mail did not match the filter after all, so the thread ended in my inbox altough the initial message of that thread actually matches subject:emacs. will keep an eye one this

[pazz]

haven't had this problem for a while now. i suspect my filters behave as i want :) closing this