enter_idle() fails to recover

[espro1]

I'm not sure if this is an issue or if it is better suited to Discussions, but the latter seems pretty inactive so I decided to share it here as well.

I've used imapfilter for a few years for firing a script when a new email is detected on the mail server. The enter_idle() function would seem to run forever: if it failed, it automatically recovered.

A few months ago imapfilter started failing every 8 hours or so when the IDLE connection dropped. I finally got around to looking into this more and I see that around the same time I estimate that this started happening, imapfilter changed how it handled recovery.

I've been trying to get enter_idle() to recover using the new functionality but I'm not making any progress. As you will be able to infer, I don't really know what I'm doing. I haven't been able to find anything in the documentation or else directly related to what I'm trying to do.

Here is what my script used to do:

while my_account.INBOX:enter_idle() do 
   os.execute("/path/to/script")
end

I just borrowed this from somewhere else but it seemed to work well enough.

I've tried a bunch of things and I'll just list a couple. Following the examples in the samples/extend.lua page:

function doidle ()
   my_account:enter_idle()
   os.execute("/path/to/script")
end

while true do
   recover (doidle)
end

Sort of combining the above with what I had earlier:

while recover(my_account.INBOX:enter_idle()) do 
   os.execute("/path/to/script
end

I read the documentation on recover again and it sort of seems like its scope does not include enter_idle(): it says recover fires when a connection is reestablished, which is exactly the thing that I need to recover (I think).

There is a good chance I am just fundamentally misunderstanding how recover() and/or enter_idle() work. As I said though I tried to look around the documentation etc. for clues but could not find anything related to this issue.

Love imapfilter by the way. Before I found it I was getting imap idle through some node.js script and it was extremely flaky. imapfilter has been absolutely solid for me.

[lefcha]

The new recover() function should work with any other method or function, even enter_idle(). The new recovery mechanism is checking whether some error was raised in Lua, and if so it cleanups the connection, and then executes again the function specified. So it doesn't care what the command does, it only cares if some there was any kind of error (networking or non-networking related).

The problem in your config is that os.execute() is never raising any error, and thus the recovery mechanism doesn't trigger. There are 2 solutions:

1. You have to do the recover() inside the executed script, so it works properly.
2. You have to manually check if script executed failed, and if it did, then raise an error to trigger recovery.

Solution (1) is straightforward, you just include all your commands in a function, and then recover() that function.

For solution (2), you should look into the return values of os.execute(), which are 3 as documented at:

https://www.lua.org/manual/5.4/manual.html#pdf-os.execute

So basically you only care for the 1st return value, which is true if the execution went well, otherwise it's nil.

So you could change your config like this:

function doidle ()
   my_account:enter_idle()
   success, _, _ = os.execute("/path/to/script")
   if not success then error() end
end

while true do
   recover (doidle)
end

This will raise an error if the execution failed, which should trigger the recovery mechanism.

Also, better include enter_idle() inside the recover mechanism, too.

[espro1]

Thank you for the response.

Interesting suggestion to check for errors on my script, I hadn't thought of that at all. Is it possible that the IMAP IDLE process is failing due to server timeout or something? That's how it looks at least superficially. E.g. if I resume after suspending the PC overnight my imapfilter process is almost always ended, which seems like a timeout thing.

My script is just a call to mbsync which syncs my local Maildir with the remote one, and it seems to work reliably when I run it manually.

I'll give this a try and see how it goes. Thanks again!

Edit: this is the error I get, by the way. I should have included this in the original message:

imapfilter: reading data through SSL; Connection reset by peer
imapfilter: idle request to [mail_server.com] failed
stack traceback:
    [C]: in function 'error'
    /usr/share/imapfilter/mailbox.lua:41: in function '_check_result'
    /usr/share/imapfilter/mailbox.lua:1070: in function 'enter_idle'
    /path/icloud.lua:51: in main chunk

[espro1]

I tried the following:

function doidle ()
   account.INBOX:enter_idle()
   success, _, _ = os.execute("mbsync -c /path/mbsyncrc ericspero:INBOX; emacsclient -e '(mu4e-update-index)'")
   if not success then error() end
end

while true do
   recover (doidle)
end

And I get this error:

gpg: AES.CFB encrypted data
gpg: encrypted with 1 passphrase
imapfilter: reading data through SSL; Connection reset by peer
imapfilter: /usr/share/imapfilter/auxiliary.lua:76: attempt to call field 'pack' (a nil value)
stack traceback:
    /usr/share/imapfilter/auxiliary.lua:76: in function 'recover'
    ...path/imapfilter/config.lua:57: in main chunk

I tried something similar with enter_idle():

function doidle () 
   success = account2.INBOX:enter_idle()
   if not success then error() end
end

while true do
   recover(doidle)
   os.execute("mbsync -c /path/mbsyncrc icloud:INBOX; emacsclient -e '(mu4e-update-index)';")
end

and get the exact same error as above.

In my previous comment I shared an error message. Does this not mean that enter_idle() is what is failing, and then not recovering? Wrapping enter_idle() in recover() does not seem to do anything.

Also I should mention that this is happening with two different IMAP servers.

[lefcha]

The error attempt to call field 'pack' (a nil value) indicates that you're using Lua 5.1 with imapfilter 2.8.0 or 2.8.1, because it triggers this known bug.

This problem with recover() and Lua 5.1, was fixed yesterday and the fix was included in version 2.8.2.

So I suggest that you:

• either upgrade to version 2.8.2, or
• use Lua 5.2/5.3/5.4 with version 2.8.0/2.8.1

[espro1]

Thanks for this - what an odd coincidence! I'm on 2.8.1, and I will probably just wait for 2.8.2 to land and will see how it goes.

[lefcha]

Until then, you could also just overwrite the auxiliary.lua file with: https://raw.githubusercontent.com/lefcha/imapfilter/7ddc109eac588acad6af59acad37d06fd1aef0d4/src/auxiliary.lua

Or just change the recover() function in the auxiliary.lua to be like this:

https://github.com/lefcha/imapfilter/blob/7ddc109eac588acad6af59acad37d06fd1aef0d4/src/auxiliary.lua#L70-L86

[espro1]

Thanks for sharing! I overwrote the recover() function with this one and then restarted the imapfilter processes, and they've been up for a few hours now. I'm getting "connection reset by peer" messages yet the the processes are still running, so it looks like it's recovering. We'll see if it's still up tomorrow morning!

[espro1]

Ok just about 24 hours later and it's still running! Woohoo!

Still can't believe what a coincidence it was that as soon as I look in to fixing this problem I've faced for months I encounter a totally unrelated bug that prevents the fix from working.

Recover seems to require that the first argument is a function (e.g. it did not like me passing it the enter_idle() call), but once I put my two calls in a function which I then pass to recover everything worked smoothly.

Thanks again for your attention and for working on this lovely tool!