Meta-Searching always searches ALL messages on server

Hi!

I am unsure if I am using imapfilter correctly but it looks like that meta-searching, while selecting the correct subset of messages, always searches all messages on the server and not the range of UID returned by the previous operation.

For example, this simple configuration, just selecting the messages since yesterday and then further selecting some mails with a specific header value:

options.timeout = 120
options.subscribe = true
options.keepalive = 5
options.starttls = true

function sort_mail()

        myaccount = IMAP {
                server = 'myserver',
                username = 'myuser',
                password = 'mypasswd',
        }

        -- Get the status of a mailbox
        myaccount.INBOX:check_status()

        mails = myaccount.INBOX:arrived_since(form_date(1))

        internallist = mails:contain_field('List-Id', 'some-internal-list')

end

sort_mail()

And this is the debuglog (I trimmed the STARTTLS and LOGIN phase) generated:

* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL C
ATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SPECIAL-
USE COMPRESS=DEFLATE
1005 OK Capability completed.

sending command (4):

1006 NAMESPACE

getting response (4):

* NAMESPACE (("" ".")) NIL NIL
1006 OK Namespace completed.

namespace (4): '' '.'
sending command (4):

1007 STATUS "INBOX" (MESSAGES RECENT UNSEEN UIDNEXT)

getting response (4):

* STATUS INBOX (MESSAGES 1894 RECENT 0 UIDNEXT 99370 UNSEEN 0)
1007 OK Status completed (0.000 secs).

sending command (4):

1008 SELECT "INBOX"

getting response (4):

* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label3 $Label1 $Label2 $Label4 $Label5 $MDNSent Junk NonJunk $Forwarded redirected $Labelwork)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Label3 $Label1 $Label2 $Label4 $Label5 $MDNSent Junk NonJunk $Forwarded redirected $Labelwork \*)] Flags permitted.
* 1894 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1051467451] UIDs valid
* OK [UIDNEXT 99370] Predicted next UID
* OK [HIGHESTMODSEQ 135505] Highest
1008 OK [READ-WRITE] Select completed (0.000 secs).
sending command (4):

1009 UID SEARCH ALL SINCE 26-May-2015

getting response (4):

* SEARCH 99325 99326 99329 99332 99334 99343 99345 99348 99349 99351 99352 99353 99354 99361 99363 99364 99368
1009 OK Search completed (0.011 secs).

sending command (4):

100A UID SEARCH ALL HEADER List-Id "some-internal-list"

getting response (4):

* SEARCH
100A OK Search completed (0.013 secs).

sending command (4):

100B LOGOUT

getting response (4):

* BYE Logging out
100B OK Logout completed.

As you can see, it first gets a list of UIDs from the "SEARCH ALL SINCE 26-May-2015" and then, when searching for the "List-Id" just uses "ALL" again instead of search only in the UIDs gathered via the first query.

From looking at the code in mailbox.lua and common.lua it seems that meta-searching only happens inside of imapfilter and is never used to optimize the queries send to the server. Which would explain why my simple example takes 15s to execute on a big INBOX.

Or I am doing something stupid and I am using imapfilter in the wrong way.

You are correct in your conclusions, but let me explain this...

At the moment, and for server requests, all messages are searched. Meta-searching right now is only useful for when you use the match_*() methods, since these methods do the matching locally, they have been tweaked to only download the relevant parts of the already matched messages, and not for all messages.

So if for example you have 1000 messages in your INBOX and 10 of them are new, and you do something like:

myaccount.INBOX:is_new():match_subject('foo')

... imapfilter will only download the Subject header of those 10 messages.

Another way that meta-searching is useful is when you already have searched something in multiple mailboxes, possibly on different accounts, and then you want to search for something additional in all of them, then meta-searching will help you write that with less effort.

But as I said, this is not very good if you want to search with more than one criteria in a huge mailbox. For this specific scenario, imapfilter has a solution (that was actually the main way of working in older versions, but it was more difficult to use, yet way more advanced), and it's the send_query() method. Basically the IMAP protocol is quite powerful and it allows you to send as many search criteria as you want in one search command, and you combine them using logical operators.

So for example in your case you could sent this command:

myaccount.INBOX:send_query('SINCE ' .. form_date(1) .. ' HEADER "List-Id" "some-internal-list"')

This way you will put minimal stress to the server and your query will probably complete much faster. You can learn more about this at RFC 3501 Section 6.4.4.

In theory it would be possible to make imapfilter send the 2nd command not for all all messages, but only for those matched by the 1st command (the IMAP protocol allows this), but I haven't looked into adding this functionality yet, partly because imapfilter already had the send_query() command. You see imapfilter's users at the time knew how to work with it to do advanced searching because imapfilter only had this way of working. Afterwards, I added all the simpler searching methods to make it easier to write config files, but as it seems this backfires in situations like yours with huge mailboxes. So I think I should reconsider this now, and investigate on implementing meta-searching to work correctly for all scenarios when I have some time...

So I think I should reconsider this now, and investigate on implementing meta-searching to work correctly for all scenarios when I have some time...

That would be really appreciated by me.

I first did the test for my current setup with a smaller INBOX and Dovecot 2.2 as server and that was very fast.

The real setup runs against an INBOX of about 3000 mails (which stay there and are not filtered) and a constant stream of about 10 messages per 5 minutes on a Courier IMAP server which is quite slow, because it has no index to accelerate searches.

So right now one filter run takes about 2 minutes to complete.

From what I understand all the needed information should already be present in the Set datastructure, so the code "just" needs to use the correct UID range for each mailbox while creating the SEARCH query instead of ALL.

Yes, I know, a really big "just" as I have no idea of how much refactoring is needed.

Yes, basically what you describe is what is needed. It sounds straightforward, but maybe there's some difficulty I don't remember, or possibly some changes are needed. Something to be seen...

Until then, have you given a try to my send_query() example? It should do what you want in one fast search query, and actually it should be even faster than the meta-searching (when it will be implemented). That's because you do one search query to the server, and you send all the info right away so the server can optimize it, instead of sending two queries with lots of UIDs on the subsequent query, etc.

I have not yet tested the raw send_query() because the filter file I have is quite big (500 lines) because I have to filter a lot of lists (by List-Id) and then filter again by subject or even body content for the mails inside each list.

Example:

        mails = account.INBOX:arrived_since(form_date(1))

        -- vbackup
        vbackup = mails:contain_field('List-Id', 'its-vbackup.lists')
        mails = mails - vbackup

        res = vbackup:contain_subject('[Success]')
        res:move_messages(account["ITS.Backup.VM.Success"])
        vbackup = vbackup - res 

        res = vbackup:contain_subject('[Warning]')
        res:move_messages(account["ITS.Backup.VM.Warning"])
        vbackup = vbackup - res 

        res = vbackup:contain_subject('[Failed]')
        res:move_messages(account["ITS.Backup.VM.Failed"])
        vbackup = vbackup - res 

        vbackup:move_messages(account["ITS.Backup.VM.Sonstiges"])
        -- /vbackup

(The code backup = vbackup - res is needed or the numbers of moved mails for the last statement are incorrect.)

I have multiple of such blocks, so changing the file will take some time to do.

I have just committed some changes that add support for narrowing down the results from previous search requests in meta-searching. You should be able to see this in action by using the -v option, and then having a look for SEARCH UID <messages> in the output.

I have done some testing, but can you give it a try and let me know how it works for you? I will release a new official version later on. Thanks!

Oh yes, it is super fast now.

The old version takes 20 seconds (on a low-load Sunday) to sift through the INBOX, even when there aren't any messages to move, the new version takes only 0.2 seconds. And it really reduces the load on the IMAP server as well.

This is a 100x improvement for me, because now something like

mails = account.INBOX:arrived_since(form_date(1))

to limit the filter to the most recent mails actually works.

Very well done, thank you!

What is the reason you limit the number of messages to 50 to use a UID based search? Something in the RFCs limiting the UID list while searching? Or did your test show problems if this list got too long?

Do you get hit by this limitation?

In the past, users got some error that the requests they were sending were too lengthy, and this was the case both for SEARCH requests, and for FLAG and COPY requests (delete and move are implemented using them, too, so basically for everything).

If you have a look at the code you can notice that when sending a FLAG or COPY request, and the messages that matched where too many, I break the items in the sequence set into multiples of 50, and send one request for each of these sets.

Also, imapfilter was based in the send_query() method in the past (it was named match() back then), and users were combining too many filters in a single command, so I was advising them to somehow break down their filters, in order to avoid this limitation in the length of the command send.

I am not sure if this limitation is still the case, or if it was/is a limitation of some servers only, but I think you can try to increase it or remove it from mailbox.lua and we can see how it works for you. You can just edit the file and imapfilter will pick up the change.

Possibly we can remove this limit, or make it user configurable...

Yes, sometimes the filtered account gets more than 50 messages in a small amount of time.

I removed the limit from my local copy and re-created several such peaks with multiple hundres messages but I could not force an error.

If the server imposes a limit I would guess it to be on the amount of UID items and not number of matched messages. (With a "UID item" being either a single UID or a range like 20:90.)

I skimmed through several IMAP RFCs and could not find any limit on the amount of UIDs one can add to a SEARCH query. Maybe the only limit is the line length of a query, but I did not look at that one.

Yes, there's nothing in the RFCs, I don't think there's a limit or something; servers should be able to handle as many items in the sequence set. Note, that in imapfilter as item of a sequence set I define something like 25 or 30:33, so the sequence set 25,30:33 would have 2 items, even though it would be acting on 5 messages.

I have done some tests on one of the server where I have a lot of messages, and I tried sending a lot of items in the sequence set in the SEARCH request, a lot more than 50, and then the same on the STORE request, and it seems that both requests were succesful and thus there is no such limitation on this server, so probably it looks like some problem of only some IMAP servers.

In this case I will make this configurable through an option, just as a workaround for people having such problems, and by default it will be set to no limit. And I will modify the STORE and FLAG requests to not be sent in multiples, unless the user has set the aforementioned option.

Looks sensible to me.

But there is on strange bug happening, if the SEARCH result is longer, the first SEARCH fails but subsequent ones work:

sending command (4):

1009 UID SEARCH ALL SINCE 30-May-2015

getting response (4):

* SEARCH 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 1151
getting response (4):

59 115365 115580 115771 116626 116781 116965 121905 121906 121907 121908
1009 OK SEARCH done.

sending command (4):

100A UID SEARCH 

getting response (4):

100A NO Error in IMAP command received by server.

sending command (4):

100B UID SEARCH UID 87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121905:121908 HEADER List-Id "its-vranger.lists"

getting response (4):

* SEARCH
100B OK SEARCH done.

The SEARCH directly after the form_date() one fails because it is empty.

The code looks like this:

mails = thm.INBOX:arrived_since(form_date(15))
vbackup = mails:contain_field('List-Id', 'its-vranger.lists')
vbackup2 = mails:contain_field('List-Id', 'its-vranger.lists')

The second contain_field query is exactly the same as the first one, but the first one fails and the second one works.

Is this something new, ie. that started happening with this new implementation of meta-searching?

This is something new with the new implementation.

Debugging further, consider the following testcase:

        mails = thm.INBOX:arrived_since(form_date(15))

        dummy = mails:contain_from('@')
        dummy = mails:contain_from('@')

        vbackup = dummy:contain_field('List-Id', 'its-vranger.lists')
        vbackup2 = dummy:contain_field('List-Id', 'its-vranger.lists')

The debuglog shows the first contain_from failing because of the empty "SEARCH UID" (see above), the seconds one works and select the same UIDs the arrived_since query already selected. At this stage, the contents of mails and dummy are the same.

Now it gets interesting:

The two dummy:contain_field queries both work. Before I made this test, i would have guessed the first one also failing, but this is not the case.

Really strange.

The very first query operating on a set from a previous SEARCH done on a mailbox seems to fail. Further operations on the same set seem to work.

OK, now I am really stumped:

Testcase:

        mails = thm.INBOX:select_all()
        dummy = mails:contain_from('@')
        dummy = mails:contain_from('@')

        mails2 = thm.INBOX:select_all()
        dummy2 = mails2:contain_from('@')
        dummy2 = mails2:contain_from('@')

The first mails:contain_from query fails. All other queries work, even the first mails2:contain_from one. If my theory of "every first query on a fresh set from a mailbox" where true, this should also fail.

But it doesn't.

This is weird.

Let me try to reproduce this, and also have a look at the code, probably something is wrong there...

Can you copy/paste here the -d debug log from the above queries you mentioned? I can't reproduce this in my server, but I suspect it has something to do with the results from the SEARCH query being sent in 2 different packets:

getting response (4):

* SEARCH 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 1151
getting response (4):

59 115365 115580 115771 116626 116781 116965 121905 121906 121907 121908
1009 OK SEARCH done.

Here are the relevant line from the testcase above:

1005 CAPABILITY

getting response (4):

* CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=PLAIN ACL ACL2=UNION
1005 OK CAPABILITY completed

sending command (4):

1006 NAMESPACE

getting response (4):

* NAMESPACE (("INBOX." ".")) NIL (("#shared." ".")("shared." "."))
1006 OK NAMESPACE completed.

namespace (4): 'INBOX.' '.'
sending command (4):

1007 STATUS "INBOX" (MESSAGES RECENT UNSEEN UIDNEXT)

getting response (4):

* STATUS "INBOX" (MESSAGES 433 RECENT 0 UIDNEXT 122183 UNSEEN 0)
1007 OK STATUS Completed.

sending command (4):

1008 SELECT "INBOX"

getting response (4):

* FLAGS ($MDNSent redirected $label2 \Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS ($MDNSent redirected $label2 \* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 433 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1087779522] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
1008 OK [READ-WRITE] Ok

sending command (4):

1009 UID SEARCH ALL

getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941
1009 OK SEARCH done.

sending command (4):

100A UID SEARCH 

getting response (4):

100A NO Error in IMAP command received by server.

sending command (4):

100B UID SEARCH UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"

getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941
100B OK SEARCH done.

sending command (4):

100C UID SEARCH ALL

getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941
100C OK SEARCH done.

sending command (4):

100D UID SEARCH UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"
getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941
100D OK SEARCH done.

sending command (4):

100E UID SEARCH UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"

getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941
100E OK SEARCH done.

sending command (4):

100F LOGOUT

getting response (4):

* BYE Courier-IMAP server shutting down
100F OK LOGOUT completed

And yes, the problem appears as soon as the SEARCH result spans more than one packet. But the resulting set seems complete, so no UIDs are missed because of that.

Hold on.

I tried the same testcase from a different computer and this time there are no problems.

There seems to be a different local problem with the one system I normally use imapfilter on.

Can you add at response.c at line 577 the following line:

debug("search response (%d):\n\n%s\n", ssn->socket, ibuf.data);

Then recompile, and run your config that fails again? Run with -d so we see the new output in the debug log.

Do you get another problem on that computer? What is that?

Ok, maybe it is too hot here right now (over 30°C) or I am just dumb. I thought I got a problem, but the system where the error does not show didn't have to edit to remove the "#mesgs > 50" part from mailbox.lua.

After making that change the error also happens. This time I also checked a different account on a different server with a different IMAP server implementation (the one at work is Courier, the other one is Dovecot 2.2).

I will now go and do the change to response.c and get back to you.

Here it is:

1005 CAPABILITY

getting response (4):

* CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=PLAIN ACL ACL2=UNION
1005 OK CAPABILITY completed

sending command (4):

1006 NAMESPACE

getting response (4):

* NAMESPACE (("INBOX." ".")) NIL (("#shared." ".")("shared." "."))
1006 OK NAMESPACE completed.

namespace (4): 'INBOX.' '.'
sending command (4):

1007 STATUS "INBOX" (MESSAGES RECENT UNSEEN UIDNEXT)

getting response (4):

* STATUS "INBOX" (MESSAGES 463 RECENT 0 UIDNEXT 122221 UNSEEN 30)
1007 OK STATUS Completed.

sending command (4):

1008 SELECT "INBOX"

getting response (4):

* FLAGS ($MDNSent redirected $label2 \Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS ($MDNSent redirected $label2 \* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 463 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1087779522] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
1008 OK [READ-WRITE] Ok

sending command (4):

1009 UID SEARCH ALL

getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941 122191 122192 122193 122194 122195 122196 122197 122198 122199 122200 122201 122202 122203 122204 122205 122206 122207 122208 122209 122210 122211 122212 122213 122214 122215 122216 122217 122218 122219 122220
1009 OK SEARCH done.

search response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 79056 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941 122191 122192 122193 122194 122195 122196 122197 122198 122199 122200 122201 122202 122203 122204 122205 122206 122207 122208 122209 122210 122211 122212 122213 122214 122215 122216 122217 122218 122219 122220
1009 OK SEARCH done.

sending command (4):

100A UID SEARCH 

getting response (4):

100A NO Error in IMAP command received by server.

search response (4):

100A NO Error in IMAP command received by server.

sending command (4):

100B LOGOUT

getting response (4):

* BYE Courier-IMAP server shutting down
100B OK LOGOUT completed

And here is the complete config-file, just to be sure:

options.timeout = 120
options.subscribe = true
options.keepalive = 5
options.starttls = true

function sort_mail()

        thm = IMAP {
                server = 'SERVER',
                username = 'USER',
                password = 'PASSWORD',
        }

        thm.INBOX:check_status()

        mails = thm.INBOX:select_all()

        dummy = mails:contain_from('@')

end

sort_mail()

Hm, that response looks fine...

Can you copy/paste the line(s) from mailbox.lua that you changed to remove the 50 items limitation?

diff --git a/src/mailbox.lua b/src/mailbox.lua
index 1d5df37..1851828 100644
--- a/src/mailbox.lua
+++ b/src/mailbox.lua
@@ -98,7 +98,7 @@ function Mailbox._send_query(self, criteria, messages)
     else
         mesgs = _extract_messages(self, messages)
     end
-    if mesgs == nil or #mesgs == 0 or #mesgs > 50 then
+    if mesgs == nil or #mesgs == 0 then
         mesgs = 'ALL'
     else
         mesgs = _make_range(mesgs)

That looks fine, too.

You can remove the like we added at responce.c, that one looks fine.

Can you change mailbox.lua at lines 104-105, so they look like this:

        print('before:', unpack(mesgs))
        mesgs = _make_range(mesgs)
        print('after:', unpack(mesgs))
        mesgs = 'UID ' .. table.concat(mesgs, ',')
        print('final:', mesgs)

Then rerun this config, and let's see what's printed on the command line.

433 messages, 0 recent, 0 unseen, in USER@SERVER/INBOX.
before: 162     19842   78878   78879   78880   78883   78884   78885   78886   78887   78889   78890   78891   78892   78893   78894   78897   78898   78899   78900   78901   78902   78903   78904   78905   78906   78907   78908   78909   78910   78911   78914   78915   78916   78917   78918   78919   78920   78921       78924   78925   78926   78927   78928   78929   78930   78931   78932   78933   78934   78935   78936   78937   78938   78939   78940   78941   78942   78943   78944   78945   78946   78947   78948   78949   78950   78953   78954   78955   78956   78957   78958   78959   78960   78961   78962   78963   78964       78965   78966   78967   78968   78969   78970   78971   78972   78973   78974   78975   78976   78977   78978   78979   78980   78981   78982   78983   78984   78985   78986   78987   78988   78989   78990   78991   78992   78993   78994   78995   78996   78997   78998   78999   79000   79001   79002   79003       79004   79005   79006   79007   79008   79009   79010   79011   79012   79013   79014   79015   79016   79017   79018   79019   79020   79021   79022   79023   79024   79025   79026   79027   79028   79029   79030   79031   79032   79033   79034   79035   79036   79037   79038   79039   79040   79041   79042       79043   79044   79045   79046   79047   79048   79049   79050   79051   79052   79053   79054   79055   79056   79057   79058   79059   79060   79061   79062   79063   79064   79065   79066   79067   79068   79069   79070   79071   79072   79073   79074   79075   79076   79077   79078   79079   79080   79081       79082   79084   79085   79086   79087   79088   79089   79090   79091   79092   79093   79094   79095   79096   79097   79098   79099   79100   79102   79103   79104   79105   79106   79107   79108   79109   79110   79111   79112   79113   79114   79115   79116   79117   79118   79119   79120   79121   79122       79123   79124   79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234   85249   85315   85316   85317   85318   85319   85320   85321   85322   85347   85350   85524   85872   85903   86068   86273   86347   87265       87534   87815   88900   89080   89299   89390   90221   90387   90701   90745   90746   90747   90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979   92990   92997   92998   93005   93202   93272   93631   93893   93894   93903   94051       95566   95740   95792   95845   95846   95847   95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665  101690  102774  103021  103347  104430  104636  104666  104715  104740  104836      104954  104955  104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355  106356  106396  106403  106407  106581  106755  106792  106817  107921  108069  108141  108142  108155  108185  108231  108312  108313  108314  108315  108318  108328  108374  108395  108396  108410      108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040  110041  110042  110043  110063  110071  110195  110234  110391  110523  110524  110646  110649  113356  113845  113883  113884  113885  113934  114143  114190  114245  114333  114362  114430  115159  115365  115580  115771      116626  116781  116965  121941
after:  162     19842   78878:78880     78883:78887     78889:78894     78897:78911     78914:78921     78924:78950     78953:79082     79084:79100     79102:79124     79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234       85249   85315:85322     85347   85350   85524   85872   85903   86068   86273   86347   87265   87534   87815   88900   89080   89299   89390   90221   90387   90701   90745:90747     90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979       92990   92997:92998     93005   93202   93272   93631   93893:93894     93903   94051   95566   95740   95792   95845:95847     95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665      101690  102774  103021  103347  104430  104636  104666  104715  104740  104836  104954:104955   104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355:106356   106396  106403  106407  106581  106755  106792  106817  107921  108069  108141:108142   108155  108185  108231      108312:108315   108318  108328  108374  108395:108396   108410  108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040:110043   110063  110071  110195  110234  110391  110523:110524   110646  110649  113356  113845  113883:113885   113934  114143  114190  114245  114333  114362      114430  115159  115365  115580  115771  116626  116781  116965  121941
final:  UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941
imapfilter: IMAP (4): 100A NO Error in IMAP command received by server.
before: 162     19842   78878   78879   78880   78883   78884   78885   78886   78887   78889   78890   78891   78892   78893   78894   78897   78898   78899   78900   78901   78902   78903   78904   78905   78906   78907   78908   78909   78910   78911   78914   78915   78916   78917   78918   78919   78920   78921       78924   78925   78926   78927   78928   78929   78930   78931   78932   78933   78934   78935   78936   78937   78938   78939   78940   78941   78942   78943   78944   78945   78946   78947   78948   78949   78950   78953   78954   78955   78956   78957   78958   78959   78960   78961   78962   78963   78964       78965   78966   78967   78968   78969   78970   78971   78972   78973   78974   78975   78976   78977   78978   78979   78980   78981   78982   78983   78984   78985   78986   78987   78988   78989   78990   78991   78992   78993   78994   78995   78996   78997   78998   78999   79000   79001   79002   79003       79004   79005   79006   79007   79008   79009   79010   79011   79012   79013   79014   79015   79016   79017   79018   79019   79020   79021   79022   79023   79024   79025   79026   79027   79028   79029   79030   79031   79032   79033   79034   79035   79036   79037   79038   79039   79040   79041   79042       79043   79044   79045   79046   79047   79048   79049   79050   79051   79052   79053   79054   79055   79056   79057   79058   79059   79060   79061   79062   79063   79064   79065   79066   79067   79068   79069   79070   79071   79072   79073   79074   79075   79076   79077   79078   79079   79080   79081       79082   79084   79085   79086   79087   79088   79089   79090   79091   79092   79093   79094   79095   79096   79097   79098   79099   79100   79102   79103   79104   79105   79106   79107   79108   79109   79110   79111   79112   79113   79114   79115   79116   79117   79118   79119   79120   79121   79122       79123   79124   79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234   85249   85315   85316   85317   85318   85319   85320   85321   85322   85347   85350   85524   85872   85903   86068   86273   86347   87265       87534   87815   88900   89080   89299   89390   90221   90387   90701   90745   90746   90747   90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979   92990   92997   92998   93005   93202   93272   93631   93893   93894   93903   94051       95566   95740   95792   95845   95846   95847   95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665  101690  102774  103021  103347  104430  104636  104666  104715  104740  104836      104954  104955  104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355  106356  106396  106403  106407  106581  106755  106792  106817  107921  108069  108141  108142  108155  108185  108231  108312  108313  108314  108315  108318  108328  108374  108395  108396  108410      108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040  110041  110042  110043  110063  110071  110195  110234  110391  110523  110524  110646  110649  113356  113845  113883  113884  113885  113934  114143  114190  114245  114333  114362  114430  115159  115365  115580  115771      116626  116781  116965  121941
after:  162     19842   78878:78880     78883:78887     78889:78894     78897:78911     78914:78921     78924:78950     78953:79082     79084:79100     79102:79124     79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234       85249   85315:85322     85347   85350   85524   85872   85903   86068   86273   86347   87265   87534   87815   88900   89080   89299   89390   90221   90387   90701   90745:90747     90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979       92990   92997:92998     93005   93202   93272   93631   93893:93894     93903   94051   95566   95740   95792   95845:95847     95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665      101690  102774  103021  103347  104430  104636  104666  104715  104740  104836  104954:104955   104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355:106356   106396  106403  106407  106581  106755  106792  106817  107921  108069  108141:108142   108155  108185  108231      108312:108315   108318  108328  108374  108395:108396   108410  108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040:110043   110063  110071  110195  110234  110391  110523:110524   110646  110649  113356  113845  113883:113885   113934  114143  114190  114245  114333  114362      114430  115159  115365  115580  115771  116626  116781  116965  121941
final:  UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941

config snippet:

        mails = thm.INBOX:select_all()

        dummy = mails:contain_from('@')
        dummy2 = mails:contain_from('@')

I added a printout for the finished query, this looks like this:

query:  ALL
before: 162     19842   78878   78879   78880   78883   78884   78885   78886   78887   78889   78890   78891   78892   78893   78894   78897   78898   78899   78900   78901   78902   78903   78904   78905   78906   78907   78908   78909   78910   78911   78914   78915   78916   78917   78918   78919   78920   78921       78924   78925   78926   78927   78928   78929   78930   78931   78932   78933   78934   78935   78936   78937   78938   78939   78940   78941   78942   78943   78944   78945   78946   78947   78948   78949   78950   78953   78954   78955   78956   78957   78958   78959   78960   78961   78962   78963   78964       78965   78966   78967   78968   78969   78970   78971   78972   78973   78974   78975   78976   78977   78978   78979   78980   78981   78982   78983   78984   78985   78986   78987   78988   78989   78990   78991   78992   78993   78994   78995   78996   78997   78998   78999   79000   79001   79002   79003       79004   79005   79006   79007   79008   79009   79010   79011   79012   79013   79014   79015   79016   79017   79018   79019   79020   79021   79022   79023   79024   79025   79026   79027   79028   79029   79030   79031   79032   79033   79034   79035   79036   79037   79038   79039   79040   79041   79042       79043   79044   79045   79046   79047   79048   79049   79050   79051   79052   79053   79054   79055   79056   79057   79058   79059   79060   79061   79062   79063   79064   79065   79066   79067   79068   79069   79070   79071   79072   79073   79074   79075   79076   79077   79078   79079   79080   79081       79082   79084   79085   79086   79087   79088   79089   79090   79091   79092   79093   79094   79095   79096   79097   79098   79099   79100   79102   79103   79104   79105   79106   79107   79108   79109   79110   79111   79112   79113   79114   79115   79116   79117   79118   79119   79120   79121   79122       79123   79124   79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234   85249   85315   85316   85317   85318   85319   85320   85321   85322   85347   85350   85524   85872   85903   86068   86273   86347   87265       87534   87815   88900   89080   89299   89390   90221   90387   90701   90745   90746   90747   90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979   92990   92997   92998   93005   93202   93272   93631   93893   93894   93903   94051       95566   95740   95792   95845   95846   95847   95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665  101690  102774  103021  103347  104430  104636  104666  104715  104740  104836      104954  104955  104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355  106356  106396  106403  106407  106581  106755  106792  106817  107921  108069  108141  108142  108155  108185  108231  108312  108313  108314  108315  108318  108328  108374  108395  108396  108410      108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040  110041  110042  110043  110063  110071  110195  110234  110391  110523  110524  110646  110649  113356  113845  113883  113884  113885  113934  114143  114190  114245  114333  114362  114430  115159  115365  115580  115771      116626  116781  116965  121941
after:  162     19842   78878:78880     78883:78887     78889:78894     78897:78911     78914:78921     78924:78950     78953:79082     79084:79100     79102:79124     79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234       85249   85315:85322     85347   85350   85524   85872   85903   86068   86273   86347   87265   87534   87815   88900   89080   89299   89390   90221   90387   90701   90745:90747     90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979       92990   92997:92998     93005   93202   93272   93631   93893:93894     93903   94051   95566   95740   95792   95845:95847     95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665      101690  102774  103021  103347  104430  104636  104666  104715  104740  104836  104954:104955   104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355:106356   106396  106403  106407  106581  106755  106792  106817  107921  108069  108141:108142   108155  108185  108231      108312:108315   108318  108328  108374  108395:108396   108410  108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040:110043   110063  110071  110195  110234  110391  110523:110524   110646  110649  113356  113845  113883:113885   113934  114143  114190  114245  114333  114362      114430  115159  115365  115580  115771  116626  116781  116965  121941
final:  UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941
query:  UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"
imapfilter: IMAP (4): 100A NO Error in IMAP command received by server.
before: 162     19842   78878   78879   78880   78883   78884   78885   78886   78887   78889   78890   78891   78892   78893   78894   78897   78898   78899   78900   78901   78902   78903   78904   78905   78906   78907   78908   78909   78910   78911   78914   78915   78916   78917   78918   78919   78920   78921       78924   78925   78926   78927   78928   78929   78930   78931   78932   78933   78934   78935   78936   78937   78938   78939   78940   78941   78942   78943   78944   78945   78946   78947   78948   78949   78950   78953   78954   78955   78956   78957   78958   78959   78960   78961   78962   78963   78964       78965   78966   78967   78968   78969   78970   78971   78972   78973   78974   78975   78976   78977   78978   78979   78980   78981   78982   78983   78984   78985   78986   78987   78988   78989   78990   78991   78992   78993   78994   78995   78996   78997   78998   78999   79000   79001   79002   79003       79004   79005   79006   79007   79008   79009   79010   79011   79012   79013   79014   79015   79016   79017   79018   79019   79020   79021   79022   79023   79024   79025   79026   79027   79028   79029   79030   79031   79032   79033   79034   79035   79036   79037   79038   79039   79040   79041   79042       79043   79044   79045   79046   79047   79048   79049   79050   79051   79052   79053   79054   79055   79056   79057   79058   79059   79060   79061   79062   79063   79064   79065   79066   79067   79068   79069   79070   79071   79072   79073   79074   79075   79076   79077   79078   79079   79080   79081       79082   79084   79085   79086   79087   79088   79089   79090   79091   79092   79093   79094   79095   79096   79097   79098   79099   79100   79102   79103   79104   79105   79106   79107   79108   79109   79110   79111   79112   79113   79114   79115   79116   79117   79118   79119   79120   79121   79122       79123   79124   79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234   85249   85315   85316   85317   85318   85319   85320   85321   85322   85347   85350   85524   85872   85903   86068   86273   86347   87265       87534   87815   88900   89080   89299   89390   90221   90387   90701   90745   90746   90747   90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979   92990   92997   92998   93005   93202   93272   93631   93893   93894   93903   94051       95566   95740   95792   95845   95846   95847   95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665  101690  102774  103021  103347  104430  104636  104666  104715  104740  104836      104954  104955  104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355  106356  106396  106403  106407  106581  106755  106792  106817  107921  108069  108141  108142  108155  108185  108231  108312  108313  108314  108315  108318  108328  108374  108395  108396  108410      108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040  110041  110042  110043  110063  110071  110195  110234  110391  110523  110524  110646  110649  113356  113845  113883  113884  113885  113934  114143  114190  114245  114333  114362  114430  115159  115365  115580  115771      116626  116781  116965  121941
after:  162     19842   78878:78880     78883:78887     78889:78894     78897:78911     78914:78921     78924:78950     78953:79082     79084:79100     79102:79124     79827   80087   80090   81407   81723   81993   82102   82286   82288   82333   83092   84073   84403   84560   85160   85170   85172   85182   85234       85249   85315:85322     85347   85350   85524   85872   85903   86068   86273   86347   87265   87534   87815   88900   89080   89299   89390   90221   90387   90701   90745:90747     90776   90783   90790   90983   91211   91548   91700   91710   92088   92352   92691   92836   92887   92939   92976   92979       92990   92997:92998     93005   93202   93272   93631   93893:93894     93903   94051   95566   95740   95792   95845:95847     95931   96000   96021   96040   96042   96116   96467   96507   98319   98490   98557   98718   99658   99817   100077  100219  100329  100331  100738  101222  101286  101420  101665      101690  102774  103021  103347  104430  104636  104666  104715  104740  104836  104954:104955   104995  105013  105063  105067  105108  105114  106037  106081  106150  106294  106332  106355:106356   106396  106403  106407  106581  106755  106792  106817  107921  108069  108141:108142   108155  108185  108231      108312:108315   108318  108328  108374  108395:108396   108410  108520  108526  108541  108693  108740  108815  109772  109934  110011  110035  110040:110043   110063  110071  110195  110234  110391  110523:110524   110646  110649  113356  113845  113883:113885   113934  114143  114190  114245  114333  114362      114430  115159  115365  115580  115771  116626  116781  116965  121941
final:  UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941
query:  UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"

I really really really don't understand why the first, totally correct query is not sent to the server while the second, identical one is sent fine. The mind boggles.

Hmm.

Could there be some kind of memory corruption going on?

I modified request.c and added a debug statement for the criteria to it:

int
request_search(session *ssn, const char *criteria, const char *charset, char
    **mesgs)
{
  int t, r;

  debug("criteria: %s\n", criteria);

  if (charset != NULL && *charset != '\0') {
    TRY(t = send_request(ssn, "UID SEARCH CHARSET \"%s\" %s",
        charset, criteria));
  } else {
    TRY(t = send_request(ssn, "UID SEARCH %s", criteria));
  }
  TRY(r = response_search(ssn, t, mesgs));

  return r;
}

Now the debuglog contains this:

criteria: UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"
sending command (4):

100A UID SEARCH H=��s1�H���

So, before TRY(t = send_request(ssn, "UID SEARCH %s", criteria)); everything seems fine, but inside send_request the criteria turns to garbage.

In addition to this, I get a segfault if i specify a charset via options.charset. I guess send_request needs a closer inspection.

Good thing you did, I was gonna ask something like that!

It seems that there is no problem with creating the query to send to the server, and also that the 2 queries are exactly identical. But from the debug log we can see that imapfilter doesn't send this query the 1st time, but instead it sends a blank query:

sending command (4):

100A UID SEARCH 

getting response (4):

100A NO Error in IMAP command received by server.

So we need to find out where does this get lost.

I was preparing debug statements for you to add in request.c buy you got me to it. Let me see your messages...

I refined my debug statements inside request_search():

int
request_search(session *ssn, const char *criteria, const char *charset, char
    **mesgs)
{
  int t, r;

  if (charset != NULL && *charset != '\0') {
    debug("criteria: %s\ncharset: %s\n", criteria, charset);
    TRY(t = send_request(ssn, "UID SEARCH CHARSET \"%s\" %s",
        charset, criteria));
  } else {
    debug("criteria: %s\n", criteria);
    TRY(t = send_request(ssn, "UID SEARCH %s", criteria));
  }
  TRY(r = response_search(ssn, t, mesgs));

  return r;
}

I also added options.charset = 'ISO-8859-1' to my config.lua to see what happens.

Output:

criteria: ALL
charset: ISO-8859-1
sending command (4):

1009 UID SEARCH CHARSET "ISO-8859-1" ALL

getting response (4):

* SEARCH 162 19842 78878 78879 78880 78883 78884 78885 78886 78887 78889 78890 78891 78892 78893 78894 78897 78898 78899 78900 78901 78902 78903 78904 78905 78906 78907 78908 78909 78910 78911 78914 78915 78916 78917 78918 78919 78920 78921 78924 78925 78926 78927 78928 78929 78930 78931 78932 78933 78934 78935 78936 78937 78938 78939 78940 78941 78942 78943 78944 78945 78946 78947 78948 78949 78950 78953 78954 78955 78956 78957 78958 78959 78960 78961 78962 78963 78964 78965 78966 78967 78968 78969 78970 78971 78972 78973 78974 78975 78976 78977 78978 78979 78980 78981 78982 78983 78984 78985 78986 78987 78988 78989 78990 78991 78992 78993 78994 78995 78996 78997 78998 78999 79000 79001 79002 79003 79004 79005 79006 79007 79008 79009 79010 79011 79012 79013 79014 79015 79016 79017 79018 79019 79020 79021 79022 79023 79024 79025 79026 79027 79028 79029 79030 79031 79032 79033 79034 79035 79036 79037 79038 79039 79040 79041 79042 79043 79044 79045 79046 79047 79048 79049 79050 79051 79052 79053 79054 79055 790
getting response (4):

56 79057 79058 79059 79060 79061 79062 79063 79064 79065 79066 79067 79068 79069 79070 79071 79072 79073 79074 79075 79076 79077 79078 79079 79080 79081 79082 79084 79085 79086 79087 79088 79089 79090 79091 79092 79093 79094 79095 79096 79097 79098 79099 79100 79102 79103 79104 79105 79106 79107 79108 79109 79110 79111 79112 79113 79114 79115 79116 79117 79118 79119 79120 79121 79122 79123 79124 79827 80087 80090 81407 81723 81993 82102 82286 82288 82333 83092 84073 84403 84560 85160 85170 85172 85182 85234 85249 85315 85316 85317 85318 85319 85320 85321 85322 85347 85350 85524 85872 85903 86068 86273 86347 87265 87534 87815 88900 89080 89299 89390 90221 90387 90701 90745 90746 90747 90776 90783 90790 90983 91211 91548 91700 91710 92088 92352 92691 92836 92887 92939 92976 92979 92990 92997 92998 93005 93202 93272 93631 93893 93894 93903 94051 95566 95740 95792 95845 95846 95847 95931 96000 96021 96040 96042 96116 96467 96507 98319 98490 98557 98718 99658 99817 100077 100219 100329 100331 100738 101222 101286 
getting response (4):

101420 101665 101690 102774 103021 103347 104430 104636 104666 104715 104740 104836 104954 104955 104995 105013 105063 105067 105108 105114 106037 106081 106150 106294 106332 106355 106356 106396 106403 106407 106581 106755 106792 106817 107921 108069 108141 108142 108155 108185 108231 108312 108313 108314 108315 108318 108328 108374 108395 108396 108410 108520 108526 108541 108693 108740 108815 109772 109934 110011 110035 110040 110041 110042 110043 110063 110071 110195 110234 110391 110523 110524 110646 110649 113356 113845 113883 113884 113885 113934 114143 114190 114245 114333 114362 114430 115159 115365 115580 115771 116626 116781 116965 121941
1009 OK SEARCH done.

criteria: UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,84560,85160,85170,85172,85182,85234,85249,85315:85322,85347,85350,85524,85872,85903,86068,86273,86347,87265,87534,87815,88900,89080,89299,89390,90221,90387,90701,90745:90747,90776,90783,90790,90983,91211,91548,91700,91710,92088,92352,92691,92836,92887,92939,92976,92979,92990,92997:92998,93005,93202,93272,93631,93893:93894,93903,94051,95566,95740,95792,95845:95847,95931,96000,96021,96040,96042,96116,96467,96507,98319,98490,98557,98718,99658,99817,100077,100219,100329,100331,100738,101222,101286,101420,101665,101690,102774,103021,103347,104430,104636,104666,104715,104740,104836,104954:104955,104995,105013,105063,105067,105108,105114,106037,106081,106150,106294,106332,106355:106356,106396,106403,106407,106581,106755,106792,106817,107921,108069,108141:108142,108155,108185,108231,108312:108315,108318,108328,108374,108395:108396,108410,108520,108526,108541,108693,108740,108815,109772,109934,110011,110035,110040:110043,110063,110071,110195,110234,110391,110523:110524,110646,110649,113356,113845,113883:113885,113934,114143,114190,114245,114333,114362,114430,115159,115365,115580,115771,116626,116781,116965,121941 FROM "@"
charset: ISO-8859-1
sending command (4):

100A UID SEARCH CHARSET "" ��(��

getting response (4):

100A NO Error in IMAP command received by server.

The search is again broken and CHARSET "" is empty, while the variable was clearly filled before send_request was executed.

Extra fun thing: if I remove the debug statement I immediately get a segfault. Seems we are looking at a Heisenbug.

I think I know what's the problem here. The problem is the charset simplifications I did along the change, and it seems that the charset variable is not set... And you just got me again :-)

Then why does it break if I set charset manually?

Full backtrace:

#0  0x00007ffff6f11e2c in _IO_vfprintf_internal (s=s@entry=0x7fffffffdb50, format=<optimized out>, format@entry=0x4114fd "UID SEARCH CHARSET \"%s\" %s", ap=ap@entry=0x7fffffffdcb8) at vfprintf.c:1642
        len = <optimized out>
        string_malloced = <optimized out>
        step0_jumps = {0, -21408, -10006, -9919, -9825, -9738, -9633, -9357, -8392, -7901, -7780, -7036, -7329, -7232, -6868, -9131, -2198, -2183, -4523, -4438, -1471, -3815, -1799, -4599, -168, -1104, -2074, -7135, -7232, -9460}
        space = 0
        is_short = 0
        use_outdigits = 0
        step1_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, -7901, -7780, -7036, -7329, -7232, -6868, -9131, -2198, -2183, -4523, -4438, -1471, -3815, -1799, -4599, -168, -1104, -2074, -7135, -7232, 0}
        group = 0
        prec = -1
        step2_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -7780, -7036, -7329, -7232, -6868, -9131, -2198, -2183, -4523, -4438, -1471, -3815, -1799, -4599, -168, -1104, -2074, -7135, -7232, 0}
        string = <optimized out>
        left = 0
        is_long_double = 0
        width = 0
        step3a_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -7133, 0, 0, 0, -6868, -9131, -2198, -2183, -4523, 0, 0, 0, 0, -4599, 0, 0, 0, 0, 0, 0}
        alt = 0
        showsign = 0
        is_long = 0
        is_char = 0
        pad = 32 ' '
        step3b_jumps = {0 <repeats 11 times>, -7329, 0, 0, -6868, -9131, -2198, -2183, -4523, -4438, -1471, -3815, -1799, -4599, -168, -1104, -2074, 0, 0, 0}
        step4_jumps = {0 <repeats 14 times>, -6868, -9131, -2198, -2183, -4523, -4438, -1471, -3815, -1799, -4599, -168, -1104, -2074, 0, 0, 0}
        is_negative = <optimized out>
        number = <optimized out>
        base = <optimized out>
        the_arg = {pa_wchar = -151967238 L'\xf6f129fa', pa_int = -151967238, pa_long_int = 140737336388090, pa_long_long_int = 140737336388090, pa_u_int = 4143000058, pa_u_long_int = 140737336388090, pa_u_long_long_int = 140737336388090, pa_double = 6.9533482996558454e-310, pa_long_double = <invalid float value>, 
          pa_string = 0x7ffff6f129fa <_IO_vfprintf_internal+22490> "\200\275(\373\377\377", 
          pa_wstring = 0x7ffff6f129fa <_IO_vfprintf_internal+22490> L"\xfb28bd80\xf00ffff\xfface985\x58b48ff\x35745a\x1600c764\xe9000000\xffffae99\xfa58bd83\x7400ffff\x30fa8332\xd0891f73\x10470349\x4108c283\xf481789\x60e900be\x48fffffe\x104bf0f\xfffe56e9\x478b49ff\x508d4808\x57894908\x83dfeb08\xfffa68bd\x497500ff\x7330fa83\x49d08936\x83104703\x894108c2\x634817\xfffe26e9\x478b49ff\x508d4808\x57894908\xf065e908\x8b49ffff\x8d480847\x89490850\x50e90857\x49ffffff\x4808478b\x4908508d\xeb085789\x30fa83c8\xd0891573\x10470349\x4108c283\xf481789\xdce900bf\x49fffffd\x4808478b\x4908508d\xeb085789\x554190e9\x53555441\x48fb8948\x2128ec81\x8f8b0000À\x850fc985ł\xc087c7\xffff0000\x8d48ffff\x1202484\x8d480000\x4820247c\x249c89\xc7000001\xe02484\xffff0000\x44c7ffff\x80042024\x8948fbad\x48482444\x40244489\x24848d48℠\x2484c748¨", pa_pointer = 0x7ffff6f129fa <_IO_vfprintf_internal+22490>, pa_user = 0x7ffff6f129fa <_IO_vfprintf_internal+22490>}
        spec = 115 's'
        _buffer = {__routine = 0x40fdc55000000000, __arg = 0x41143d, __canceltype = 14, __prev = 0x67}
        _avail = <optimized out>
        thousands_sep = 0x0
        grouping = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>
        done = 25
        f = 0x411516 "s"
        lead_str_end = <optimized out>
        end_of_spec = <optimized out>
        work_buffer = " \000\000\000\000\000\000\000\020\000\000\000\060\000\000\000\220\335\377\377\377\177\000\000\320\334\377\377\377\177\000\000\240\335\377\377\377\177\000\000\340\334\377\377\377\177\000\000'\334\377\377\377\177", '\000' <repeats 14 times>, "\377\177", '\000' <repeats 26 times>, "\377\377\377\377\377\377\377\377\372)\361\366\377\177\000\000\000\000\000\000\000\000\000\000\372)\361\366\377\177\000\000\377\377\377\377\377\377\377\377U\024A", '\000' <repeats 13 times>, "\001\000\000\000\377\177\000\000U\024A\000\000\000\000\000U\024A\000\000\000\000\000n\000\000\000\000\000\000\000\001\000\000\000\060\000\000\000U\024A\000\000\000\000\000"...
        workstart = 0x0
        workend = 0x7fffffffdb08 "P\333\377\377\377\177"
        ap_save = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffffffddc0, reg_save_area = 0x7fffffffdcd0}}
        nspecs_done = 1
        save_errno = <optimized out>
        readonly_format = 0
        args_malloced = 0x0
        specs = <optimized out>
        specs_malloced = false
        jump_table = "\001\000\000\004\000\016\000\006\000\000\a\002\000\003\t\000\005\b\b\b\b\b\b\b\b\b\000\000\000\000\000\000\000\032\000\031\000\023\023\023\000\035\000\000\f\000\000\000\000\000\000\025\000\000\000\000\022\000\r\000\000\000\000\000\000\032\000\024\017\023\023\023\n\017\034\000\v\030\027\021\026\f\000\025\033\020\000\000\022\000\r"
        __PRETTY_FUNCTION__ = "_IO_vfprintf_internal"
#1  0x00007ffff6f39459 in _IO_vsnprintf (string=0x73e2d5 "UID SEARCH CHARSET \" Jn\" 859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,8228"..., maxlen=<optimized out>, 
    format=format@entry=0x4114fd "UID SEARCH CHARSET \"%s\" %s", args=args@entry=0x7fffffffdcb8) at vsnprintf.c:119
        sf = {f = {_sbf = {_f = {_flags = -72515583, _IO_read_ptr = 0x73e2d5 "UID SEARCH CHARSET \" Jn\" 859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,8228"..., 
                _IO_read_end = 0x73e2d5 "UID SEARCH CHARSET \" Jn\" 859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,8228"..., 
                _IO_read_base = 0x73e2d5 "UID SEARCH CHARSET \" Jn\" 859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,8228"..., 
                _IO_write_base = 0x73e2d5 "UID SEARCH CHARSET \" Jn\" 859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,8228"..., 
                _IO_write_ptr = 0x73e2ee "859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073"..., _IO_write_end = 0x73eace "", 
                _IO_buf_base = 0x73e2d5 "UID SEARCH CHARSET \" Jn\" 859-1\" UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,8228"..., _IO_buf_end = 0x73eace "", _IO_save_base = 0x0, 
                _IO_backup_base = 0x0, _IO_save_end = 0x0, _markers = 0x0, _chain = 0x0, _fileno = 1040, _flags2 = 0, _old_offset = 140737339893280, _cur_column = 0, _vtable_offset = 97 'a', _shortbuf = "", _lock = 0x0, _offset = 1040, _codecvt = 0x7ffff6f42710 <_int_realloc+560>, _wide_data = 0xffffffffffffffff, 
                _freeres_list = 0x0, _freeres_buf = 0x7ffff726a620 <main_arena>, _freeres_size = 6387440, _mode = -1, _unused2 = "\000\000\000\000\020\b\000\000\000\000\000\000\001\b\000\000\000\000\000"}, vtable = 0x7ffff7269460 <_IO_strn_jumps>}, _s = {_allocate_buffer = 0x0, 
              _free_buffer = 0x7ffff7bbe7ff <luaS_newlstr+383>}}, overflow_buf = "P\005\000\000\000\b\000\000\220Oa\000\000\000\000\000R\005\000\000\000\000\000\000\375\024A\000\000\000\000\000\002\000\000\000\000\000\000\000\n\020\000\000\000\000\000\000\005\000\000\000\000\000\000\000\267n@\000\000\000\000"}
        ret = <optimized out>
#2  0x0000000000408567 in send_request (ssn=0x65e120, fmt=fmt@entry=0x4114fd "UID SEARCH CHARSET \"%s\" %s") at request.c:73
        n = <optimized out>
        args = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7fffffffddc0, reg_save_area = 0x7fffffffdcd0}}
        t = 4106
#3  0x0000000000409dc9 in request_search (ssn=0x65e120, criteria=criteria@entry=0x73dd88 "UID 162,19842,78878:78880,78883:78887,78889:78894,78897:78911,78914:78921,78924:78950,78953:79082,79084:79100,79102:79124,79827,80087,80090,81407,81723,81993,82102,82286,82288,82333,83092,84073,84403,"..., 
    charset=charset@entry=0x6b4898 "ISO-8859-1", mesgs=mesgs@entry=0x7fffffffdde8) at request.c:412
        t = <optimized out>
        r = <optimized out>
#4  0x0000000000404967 in ifcore_search (lua=0x65dda0) at core.c:379
        r = <optimized out>
        mesgs = 0x0
#5  0x00007ffff7bb5c3d in luaD_precall (L=L@entry=0x65dda0, func=func@entry=0x739a40, nresults=nresults@entry=2) at ldo.c:319
        f = 0x4048b7 <ifcore_search>
        ci = <optimized out>
        n = <optimized out>
        funcr = <optimized out>
#6  0x00007ffff7bc159d in luaV_execute (L=L@entry=0x65dda0) at lvm.c:709
        b = <optimized out>
        nresults = 2
        i = <optimized out>
        ra = 0x739a40
        ci = <optimized out>
        cl = 0x697fe0
        k = 0x6670e0
        base = <optimized out>
#7  0x00007ffff7bb5fa8 in luaD_call (L=0x65dda0, func=<optimized out>, nResults=<optimized out>, allowyield=<optimized out>) at ldo.c:402
No locals.
#8  0x00007ffff7bb55bf in luaD_rawrunprotected (L=L@entry=0x65dda0, f=f@entry=0x7ffff7bb0bd0 <f_call>, ud=ud@entry=0x7fffffffe060) at ldo.c:131
        oldnCcalls = 0
        lj = {previous = 0x0, b = {{__jmpbuf = {6675872, 673945611725098320, 0, 1, 1, 6676000, -674319525611516592, -674337629140497072}, __mask_was_saved = 0, __saved_mask = {__val = {969273691, 4199387, 4294967295, 3199363768272845170, 140737349594152, 140737354101968, 7280677323197079923, 7164793138308605038, 
                  7593435248410454388, 6877956704234857068, 2910562809780397412, 7308533145361212787, 732243293787747187, 8463501140575658016, 3343478956348173938, 7310874224626656627}}}}, status = 0}
#9  0x00007ffff7bb6201 in luaD_pcall (L=L@entry=0x65dda0, func=func@entry=0x7ffff7bb0bd0 <f_call>, u=u@entry=0x7fffffffe060, old_top=32, ef=<optimized out>) at ldo.c:603
        status = <optimized out>
        old_ci = 0x65de20
        old_allowhooks = 1 '\001'
        old_nny = 1
        old_errfunc = 0
#10 0x00007ffff7bb2186 in lua_pcallk (L=0x65dda0, nargs=nargs@entry=0, nresults=nresults@entry=-1, errfunc=errfunc@entry=-2, ctx=ctx@entry=0, k=k@entry=0x0) at lapi.c:949
        c = {func = 0x65e090, nresults = -1}
        status = <optimized out>
        func = <optimized out>
#11 0x0000000000406e37 in start_lua () at lua.c:86
No locals.
#12 0x0000000000405e47 in main (argc=5, argv=0x7fffffffe1b8) at imapfilter.c:128
        c = <optimized out>

Hm, indeed ,this is quite strange... If options.charset is not defined then it should be set to '' and that should be passed to ifcore.search(). But then while it was set to ISO-8859-1 it gets back to blank, and the criteria are corrupted.

It looks like the problem is there in send_request() and data are not correctly written to the obuf. I'm having a look there now...

Ok, I tried setting query inside _send_query() to your query and see what happens, and I get a segfault. So I think this is related to the size of the request to be sent, and writing that to `obuf, and since I can reproduce it I think I'll have a solution soon (unless you find it first again)...

Well, this is as far as my knowledge of C goes, sorry. I had a quick look at send_request and my head started spinning from all the obuf here and obuf there. I am a Perl guy, pointer logic inside C scares me. ;)

Great, you left something for me to do! :-)

Hmm, obuf.size seems to be limited to 1024 bytes. Seems suspicous.

This is reallocated on a needed basis. I found the problem, it had to do with va_start() and vsnprintf(). Can you pull my last commit, compile and try again?

Yes, this looks alright now.

Great! Thanks for all your efforts, and your help, I'm glad we finally nailed it!

Let me know if you have any further questions, and if nothing comes up I will go ahead and make this an official release.

Let me just quickly run this through my Debian builder, deploy it to the main system running imapfilter and test there again. Should only take about 10 minutes.

Ah, the repository still contains the "#mesgs > 50" bit. Do you want to take that out or do your STORE and FLAG optimizations first?

I just pushed that change, too.

Everything is fine here, it passed all my tests, so: good to go.

Ah, I spoke to soon.

If you don't set options.sequenceset in your config.file, you get a /usr/share/imapfilter/mailbox.lua:102: attempt to compare number with nil error.

Damn that got there by mistake, I didn't want to add support for that yet... Let me fix that.

lefcha / imapfilter

Meta-Searching always searches ALL messages on server #93