Need some example on how to use `IDLE`

[PHProger-themus]

Trying to write NestJS Websocket new emails observer, i.e. I need to get all new emails every 10 seconds.

I don't really understand how IDLE works. I've read that it can be invoked and then I need to send DONE command if I want to reissue IDLE (reissuing every 29 minutes according to RFC), but there's no way to either send command or invoke some done() method in this library. If I'm using auto idle with maxIdleTime = 10000, some new messages are not being sent to me from server, what can be wrong? My code is:

import { Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { ImapFlow } from 'imapflow';

@Injectable()
export class ImapService {
  private readonly imap: ImapFlow;

  constructor(
    private readonly config: ConfigService
  ) {
    this.imap = new ImapFlow({
      host: this.config.get<string>('IMAP_HOST'),
      port: this.config.get<number>('IMAP_PORT'),
      auth: {
        user: this.config.get<string>('IMAP_USERNAME'),
        pass: this.config.get<string>('IMAP_PASSWORD'),
      },
      secure: true,
      maxIdleTime: 10000
    });
  }

  async connect() {
    await this.imap.connect();
  }

  async subscribe(folder: string) {
    await this.imap.getMailboxLock(folder);
    this.imap.on('exists', async () => {
      let message = await this.imap.fetchOne('*', { source: true });
      console.log(message.source.toString());
    });
  }

  async disconnect() {
    await this.imap.logout();
  }
}

This method is returning only 1 new message and then repeatedly reissuing IDLE without further observing. Maybe 10 seconds is too small interval? I know that I need to use fetch() instead of fetchOne() to get all the new messages and not only last one each time, but now I need to make it work properly at least...

[benbucksch]

My understanding is that this library will issue IDLE automatically, unless you explicitly disable it. I.e. you don't need to issue it manually.

this.imap.on('exists', async () => {

You do need to attach event listeners. You are already subscribing to exists event in your code above - that's good. You'll need to read out the info that you get with that event, and react accordingly.

There also other events that you'll want to attach, e.g. "flags", which tells you about read state changes, and "expunge", which tells you about deleted messages.

maxIdleTime = 10000 Maybe 10 seconds is too small interval?

Yes, re-issueing IDLE every 10 seconds seems awfully short. I'd use at least 30 seconds, maybe even 5 or 10 minutes.

[benbucksch]

await this.imap.getMailboxLock(folder);

I don't know why you're locking the mailbox before subscribing to events, immediately after login. That seems like it might break things.

[vid]

I'm also struggling with this, for flags changes, I too can get events to fire once, then it's not clear what's happening. The awaitd idle returns, so await the next one in the loop, but it doesn't see successive flags events. It doesn't seem to matter if I getMailboxLock before or after setting up the events.

A working example would really be appreciated.

    async idleLoop() {
        const client = await this.controller.getClient(); // default client with disableAutoIdle since we are using idle()
        const now = new Date();
        const lock = await client.getMailboxLock(this.mailbox);
        const connection = { client, lock };

        client.on('exists', async (msgCount: number) => {
            console.log(`New message in ${this.mailbox}:`, msgCount);
            this.handleNewMessage(connection, this.mailbox, msgCount);
        });

        client.on('expunge', async ({ seq }: { seq: number; path: string; vanished: boolean }) => {
            console.log(`Message deleted in ${this.mailbox}:`, seq);
            this.handleMessageDeletion(connection, this.mailbox, seq);
        });

        client.on('flags', async ({ seq, flags }: { seq: number; flags: Set<string> }) => {
            console.log(`Message flags updated in ${this.mailbox}:`, seq, flags);
            this.handleFlagUpdate(connection, this.mailbox, seq, flags);
        });

        try {
            while (true) {
                console.log('idling', this.mailbox);
                await client.idle();
                console.log(`finished idle for ${this.mailbox} in ${new Date().getTime() - now.getTime()}ms`);
            }
        } catch (error) {
            console.error(`Error monitoring mailbox ${this.mailbox}:`, error);
        } finally {
            console.log(`finalize monitor for ${this.mailbox}`);
            await this.controller.closeClient(connection);
        }
    }

[benbucksch]

1. As the docs say, IDLE is issued automatically by default (unless you explicitly disable that), so you do not need to issue IDLE yourself. The lib does it for you.

https://imapflow.com/module-imapflow-ImapFlow.html ImapFlow Constructor docs say: "disableAutoIdle Boolean false if true then IDLE is not started automatically. Useful if you only need to perform specific tasks over the connection" (Translation into English, because this is unfortunately double negation: disableAutoIdle is default false, i.e. autoidle is on by default, i.e. IDLE is done automatically by default.)

2. Do not lock the mailbox. (IDLE will probably not work while the lock is in place.)
3. Attach the event listeners only once after you open the connection, not in a loop as you do.

Sample code is simple and straight forward:

let options = { ... } // do *not* set disableAutoIdle here
let connection = new ImapFlow(options);
await connection.connect();
connection.on("close", async () => { ... });
connection.on("error" async () => { ... });
connection.on("exists", async (info) => {
  try {
    let folder = getFolderByPath(info.path);
    ... info.count
    ... info.prevCount
  } catch (ex) { error(ex); }
});
connection.on("flags", async (info) => { ... });
connection.on("expunge", async (info) => { ... });
....
// issue IMAP commands normally, without regard for IDLE
conn.mailboxOpen("INBOX");

[vid]

Thank you for that clarification.

I've based the following minimal example on your code. The only change is wrapping event listening in a promise and adding process SIGINT (I think imapflow does that anyway, it's just to show it is working). The intent is a standalone monitor that will use its own client to handle events. Unfortunately, it doesn't see any events except SIGINT (my previous code saw the first flags event, plus other events). Should this work for all mailboxes? Thank you.

        const client = new ImapFlow({
            host: mailHost!,
            port: 993,
            secure: true,
            auth: {
                user: user!,
                pass,
            },
        });
        await client.connect();
        await new Promise<void>((resolve, reject) => {
            console.log('Waiting for imap events');
            client.on('close', async () => {
                console.log('close');
            });
            client.on('error', async (info) => {
                console.log('error', info);
            });
            client.on('exists', async (info) => {
                console.log('exists', info);
            });
            client.on('flags', async (info) => {
                console.log(`flags`, info);
            });
            client.on('expunge', async (info) => {
                console.log(`info`, info);
            });
            process.on('SIGINT', async () => {
                console.log('Process interrupted, closing IMAP client');
                try {
                    await client.logout();
                    resolve();
                } catch (error) {
                    reject(error);
                }
            });
        });

imap logs, from connection:

{"level":20,"time":1721574492318,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"2 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in","cid":"1hf6qc2frh1arri4hlbc"}
{"level":30,"time":1721574492318,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"auth","msg":"User authenticated","cid":"1hf6qc2frh1arri4hlbc","user":"me@place.org"}
{"level":20,"time":1721574492319,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"3 NAMESPACE","cid":"1hf6qc2frh1arri4hlbc"}
{"level":20,"time":1721574492412,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"* NAMESPACE ((\"\" \".\")) NIL NIL","cid":"1hf6qc2frh1arri4hlbc"}
{"level":20,"time":1721574492413,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"3 OK Namespace completed (0.001 + 0.000 secs).","cid":"1hf6qc2frh1arri4hlbc"}
{"level":20,"time":1721574492413,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"4 ENABLE CONDSTORE","cid":"1hf6qc2frh1arri4hlbc"}
{"level":20,"time":1721574492505,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"* ENABLED CONDSTORE","cid":"1hf6qc2frh1arri4hlbc"}
Waiting for imap events
{"level":20,"time":1721574492506,"pid":758611,"hostname":"wupwup","component":"imap-connection","cid":"1hf6qc2frh1arri4hlbc","src":"s","msg":"4 OK Enabled (0.001 + 0.000 secs).","cid":"1hf6qc2frh1arri4hlbc"}

[benbucksch]

Should this work for all mailboxes?

No, IMAP can only monitor one folder at a time. Typically, you watch the INBOX. You have to select the folder to be watched (IMAP SELECT command) using mailboxOpen() or getMailboxLock() (but I think you'll have to release the lock again), like:

conn.mailboxOpen("INBOX");
await conn.fetch(...);

or:

let lock;
try {
  lock = conn.getMailboxLock("INBOX");
  await conn.fetch(...);
} finally {
  lock?.release();
}

I've also adapted my minimal code above accordingly.

[benbucksch]

Also, some servers do not support IDLE, but I'd hope they are rare these days. Most common servers from the last 15 years or so support it.

[andris9]

As already stated in other comments, you do not have to use the idle method explicitly. Select a mailbox, and ImapFlow will start idling automatically after some time and cancel the idle once you want to do something else. If the server does not support the IDLE command, then ImapFlow uses NOOP-loop instead.

[andris9]

Just to point out - ImapFlow was extracted from EmailEngine into a separate open-source library, and a lot of usage patterns make mainly sense in EmailEngine's context. As ImapFlow continues to power EmailEngine under the hood, this will stay the same way as well (e.g., I don't plan to change any methods, etc., if it breaks anything in EmailEngine or to add new methods that EmailEngine does not need)

[vid]

@benbucksch thank you very much for that info, it was what I needed.

@andris9 I appreciate that and thank you for the imapflow library, this is for a non-commercial experimental project.

[PHProger-themus]

I didn't write here for some time, was a little busy. Thank you @benbucksch for your info, I'll try it out soon and either mark your answer with thumb up or ask further questions if something will go wrong again. But I'm pretty much sure that the problem is in mailbox lock and small re-issuing interval as you've mentioned. The thing is I've never worked with IMAP before and some concepts remained unclear to me.

[benbucksch]

I've never worked with IMAP before and some concepts remained unclear to me.

Fair enough. IMAP can be hard. This lib makes it fairly easy, but of course the basic ideas have to be there.

For reference, here's the current IMAP4rev2 standard (RFC 9051). I'll link the relevant sections. It's just a few paragraphs to read, but very helpful.

• IDLE command
• SELECT command
• EXISTS server response