IDLE check response hangs after a few hours

[seanthegeek]

I'm trying to use imapclient's IDLE functionality to monitor an inbox hosted in Office 365. I get responses back for the first hour or two of the IDLE session, then something hangs, and pressing ^c results in a BrokenPipeError exception as the client tries to send DONE. Any idea what is wrong and how to fix it?

class IMAPError(RuntimeError):
    """Raised when an IMAP error occurs"""

def watch_inbox(host, username, password, callback, reports_folder="INBOX",
                archive_folder="Archive", delete=False, test=False, wait=30,
                nameservers=None, dns_timeout=6.0):
    """
    Use an IDLE IMAP connection to parse incoming emails, and pass the results
    to a callback function

    Args:
        host: The mail server hostname or IP address
        username: The mail server username
        password: The mail server password
        callback: The callback function to receive the parsing results
        reports_folder: The IMAP folder where reports can be found
        archive_folder: The folder to move processed mail to
        delete (bool): Delete  messages after processing them
        test (bool): Do not move or delete messages after processing them
        wait (int): Number of seconds to wait for a IMAP IDLE response
        nameservers (list): A list of one or more nameservers to use
        (8.8.8.8 and 4.4.4.4 by default)
        dns_timeout (float): Set the DNS query timeout
    """
    rf = reports_folder
    af = archive_folder
    ns = nameservers
    dt = dns_timeout

    server = imapclient.IMAPClient(host)
    server.login(username, password)

    server.select_folder(rf)

    # Start IDLE mode
    server.idle()

    while True:
        try:
            responses = server.idle_check(timeout=wait)
            if responses is not None:
                for response in responses:
                    if response[1] == b'RECENT' and response[0] > 0:
                        res = get_dmarc_reports_from_inbox(host, username,
                                                           password,
                                                           reports_folder=rf,
                                                           archive_folder=af,
                                                           delete=delete,
                                                           test=test,
                                                           nameservers=ns,
                                                           dns_timeout=dt)
                        callback(res)
                        break
        except imapclient.exceptions.IMAPClientError as error:
            error = error.__str__().lstrip("b'").rstrip("'").rstrip(".")
            raise IMAPError(error)
        except socket.gaierror:
            raise IMAPError("DNS resolution failed")
        except ConnectionRefusedError:
            raise IMAPError("Connection refused")
        except ConnectionResetError:
            raise IMAPError("Connection reset")
        except ConnectionAbortedError:
            raise IMAPError("Connection aborted")
        except TimeoutError:
            raise IMAPError("Connection timed out")
        except ssl.SSLError as error:
            raise IMAPError("SSL error: {0}".format(error.__str__()))
        except ssl.CertificateError as error:
            raise IMAPError("Certificate error: {0}".format(error.__str__()))
        except KeyboardInterrupt:
            break

    try:
        server.idle_done()
        server.logout()
    except BrokenPipeError:
        pass

[NicolasLM]

A connection in IDLE mode must be reactivated periodically, the RFC requires at least every 29 minutes. I personally found 13 minutes to be the sweet spot.

[seanthegeek]

Ah. It would be great if imapclient handled that for you, being a higher-level library. Do you have any example of how to manage that? A threaded timer?

I just checked the docs again and noticed

Note that IMAPClient does not handle low-level socket errors that can happen when maintaining long-lived TCP connections. Users are advised to renew the IDLE command every 10 minutes to avoid the connection from being abruptly closed.

D'oh!

[NicolasLM]

See #324 for an example on how to IDLE and renew many connections at once.

In your case as you appear to have a single connection, you can do something very simple:

import time

start_time = time.monotonic()
responses = server.idle_check(timeout=wait)
if time.monotonic() - start_time > 13*60:
    server.idle_done()
    server.idle()

[seanthegeek]

Thanks!