Meta-Searching always searches ALL messages on server

[shartge]

Hi!

I am unsure if I am using imapfilter correctly but it looks like that meta-searching, while selecting the correct subset of messages, always searches all messages on the server and not the range of UID returned by the previous operation.

For example, this simple configuration, just selecting the messages since yesterday and then further selecting some mails with a specific header value:

options.timeout = 120
options.subscribe = true
options.keepalive = 5
options.starttls = true

function sort_mail()

        myaccount = IMAP {
                server = 'myserver',
                username = 'myuser',
                password = 'mypasswd',
        }

        -- Get the status of a mailbox
        myaccount.INBOX:check_status()

        mails = myaccount.INBOX:arrived_since(form_date(1))

        internallist = mails:contain_field('List-Id', 'some-internal-list')

end

sort_mail()

And this is the debuglog (I trimmed the STARTTLS and LOGIN phase) generated:

* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL C
ATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SPECIAL-
USE COMPRESS=DEFLATE
1005 OK Capability completed.

sending command (4):

1006 NAMESPACE

getting response (4):

* NAMESPACE (("" ".")) NIL NIL
1006 OK Namespace completed.

namespace (4): '' '.'
sending command (4):

1007 STATUS "INBOX" (MESSAGES RECENT UNSEEN UIDNEXT)

getting response (4):

* STATUS INBOX (MESSAGES 1894 RECENT 0 UIDNEXT 99370 UNSEEN 0)
1007 OK Status completed (0.000 secs).

sending command (4):

1008 SELECT "INBOX"

getting response (4):

* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label3 $Label1 $Label2 $Label4 $Label5 $MDNSent Junk NonJunk $Forwarded redirected $Labelwork)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label3 $Label1 $Label2 $Label4 $Label5 $MDNSent Junk NonJunk $Forwarded redirected $Labelwork \*)] Flags permitted.
* 1894 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1051467451] UIDs valid
* OK [UIDNEXT 99370] Predicted next UID
* OK [HIGHESTMODSEQ 135505] Highest
1008 OK [READ-WRITE] Select completed (0.000 secs).
sending command (4):

1009 UID SEARCH ALL SINCE 26-May-2015

getting response (4):

* SEARCH 99325 99326 99329 99332 99334 99343 99345 99348 99349 99351 99352 99353 99354 99361 99363 99364 99368
1009 OK Search completed (0.011 secs).

sending command (4):

100A UID SEARCH ALL HEADER List-Id "some-internal-list"

getting response (4):

* SEARCH
100A OK Search completed (0.013 secs).

sending command (4):

100B LOGOUT

getting response (4):

* BYE Logging out
100B OK Logout completed.

As you can see, it first gets a list of UIDs from the "SEARCH ALL SINCE 26-May-2015" and then, when searching for the "List-Id" just uses "ALL" again instead of search only in the UIDs gathered via the first query.

From looking at the code in mailbox.lua and common.lua it seems that meta-searching only happens inside of imapfilter and is never used to optimize the queries send to the server. Which would explain why my simple example takes 15s to execute on a big INBOX.

Or I am doing something stupid and I am using imapfilter in the wrong way.

[lefcha]

Please, try again now, thanks!

[shartge]

if mesgs == nil or #mesgs == 0 or then

Please remove the or before then.

[shartge]

Thank you, works fine now.

[lefcha]

Damn, this isn't my night (and it's like 31°C here)...

[shartge]

Not much cooler here. But I think we found and you fixed the problems. And improved the filtering quite a bit. This issue can be closed, I think.

[lefcha]

I agree!

[lefcha]

Closed by f6e68bf62000b0c45b78b042fe651fe686a46938 and later commits.