Support RFC5819 (LIST-STATUS)

rcubetrac commented 14 years ago

Reported by brandond on 15 Oct 2010 08:37 UTC as Trac ticket #1487058

RFC 5819 adds support for extended information (total and unread messages, in the most common case) to be returned in the LIST and LSUB commands. This removes the need to LIST/LSUB and then subsequently retrieve the STATUS of each mailbox.

http://www.ietf.org/rfc/rfc5819.txt

Migrated-From: http://trac.roundcube.net/ticket/1487058

rcubetrac commented 14 years ago

Milestone changed by @alecpl on 15 Oct 2010 10:02 UTC

0.5-beta => later

rcubetrac commented 14 years ago

Comment by brandond on 17 Oct 2010 09:12 UTC

I wasn't aware that the current IMAP engine doesn't make use of the IMAP4rev1 STATUS command at all - for things like unread message count it actually does a SEARCH ALL UNSEEN and then returns the size of the result list.

I suppose I could open another ticket for Roundcube to use STATUS where possible; perhaps that could be added at the same time as LIST-EXTENDED support. The results from the LIST-EXTENDED versions of LIST/LSUB could be cached; countUnseen (and any other inefficient routines) could be modified to use STATUS instead of SEARCH when cached information is not available.

rcubetrac commented 14 years ago

Comment by brandond on 17 Oct 2010 12:06 UTC

The patch I just attached adds support for the IMAP4rev1 STATUS command, and (if available) uses it to find unread message counts. On my test system, this executes in about half the time as the current SEARCH method.

We could save a bit of additional work if we used the LIST-EXTENDED extension to precache the data in _listMailboxes, and then used this instead of calling STATUS on each folder. I'll take a look at doing that, although I'm not sure at this point how to go about caching, and whether or not it should be done unconditionally.

rcubetrac commented 14 years ago

Comment by @alecpl on 17 Oct 2010 12:36 UTC

Another related extension is ESEARCH (RFC4731). RFC3501 says: "Unlike the LIST command, the STATUS command is not guaranteed to be fast in its response. Under certain circumstances, it can be quite slow." What server are you using? I'd like to know what would be the performance of "SEARCH RETURN (COUNT) UNSEEN" in comparision with STATUS and with basic SEARCH.

rcubetrac commented 14 years ago

Comment by brandond on 17 Oct 2010 19:58 UTC

Yeah, SELECT is not guaranteed to be fast, but neither is SEARCH. I'm using Dovecot, which does optimize flag searches pretty well, but I believe that STATUS is even further optimized - Timo is co-author of RFC5819, and recognizes that retrieving STATUS for a large number of folders is common client behavior; see section 2.

I think most of the savings over the current implementation are just on the wire - I have a number of folders with thousands of unread messages, and just receiving and parsing the UID list is time-consuming.

I'll see if I can get some timing data on STATUS vs ESEARCH, but it should be noted that ESEARCH requires an additional round trip to the server to SELECT the folder before SEARCHing. For Roundcube this means we verify that the folder was selected successfully, parse and store the STATUS response fields, etc, all even before the SEARCH can begin - whereas STATUS results are returned and parsed immediately.

rcubetrac commented 14 years ago

Comment by brandond on 17 Oct 2010 23:11 UTC

I asked Timo for his input on STATUS versus SEARCH, he says:

The reason why STATUS is mentioned to be possibly slow is to discourage clients from doing a STATUS to all mailboxes. STATUS is definitely faster than SELECT+SEARCH with all IMAP servers.

rcubetrac commented 14 years ago

Comment by @alecpl on 18 Oct 2010 06:18 UTC

Thanks for the info, we'll probably use STATUS then, but we should check for LIST-EXTENDED capability, because if this isn't supported there's no sense to call STATUS and then SELECT+SEARCH. Still ESEARCH would be useful when when skip_deleted=true or STATUS doesn't return UNSEEN (for any reason).

rcubetrac commented 14 years ago

Milestone changed by @alecpl on 18 Oct 2010 06:18 UTC

later => 0.5-beta

rcubetrac commented 14 years ago

Comment by brandond on 18 Oct 2010 07:07 UTC

I think I may have confused the issue by getting into STATUS; STATUS is IMAP4rev1 core while LIST-EXTENDED and ESEARCH are IMAP4rev1 extensions. As I see it:

LIST-EXTENDED could be used to populate icache for EXISTS/UNSEEN counts during LIST/LSUB for later use by some cases of messagecount.
STATUS could be used for servers that don't support LIST-EXTENDED or when data is not in icache
SELECT+ESEARCH could be used for more complicated scenarios (skip_deleted, etc)
SELECT+SEARCH or SELECT+THREAD is fallback for older servers or more complicated cases

Does that sound right?

rcubetrac commented 14 years ago

Comment by @alecpl on 18 Oct 2010 07:35 UTC

Right. My mistake in last comment. I've miss that UNSEEN is standard (RFC3501) STATUS response, so checking for LIST-EXTENDED before using STATUS is not needed. There is a place in RFC3501 saying that UNSEEN is for "number of the first message without the \Seen flag set" (see 7.1), but STATUS command response is described as number of messages. This is what I've miss.

rcubetrac commented 14 years ago

Comment by @alecpl on 20 Oct 2010 13:33 UTC

Patch with small mods added in 710e2748. I'll work on ESEARCH support.

rcubetrac commented 14 years ago

Comment by @alecpl on 20 Oct 2010 18:43 UTC

ESEARCH support added in 659cf14c.

rcubetrac commented 14 years ago

Comment by @alecpl on 22 Oct 2010 11:16 UTC

I've added LIST-STATUS support to rcube_imap_generic class in 6a16031d. Using it will require a lot more changes. Currently I don't have a test IMAP server supporting LIST-STATUS.

rcubetrac commented 14 years ago

Milestone changed by @alecpl on 22 Oct 2010 11:16 UTC

0.5-beta => later

rcubetrac commented 14 years ago

Summary changed by @alecpl on 22 Oct 2010 11:16 UTC

Support RFC5819 (LIST-EXTENDED)

Support RFC5819 (LIST-STATUS)

rcubetrac commented 14 years ago

Comment by brandond on 23 Oct 2010 09:36 UTC

I guess I'm unclear what the difference is between LIST-STATUS and LIST-EXTENDED. I suppose I should re-read the RFC.

Anyway, I've got a server set up with Dovecot 2.0.5 and an account for you. I'll e-mail you privately with login information. I also set up a shared public namespace, as I seem to recall that was something that was awaiting further testing as well.

rcubetrac commented 14 years ago

Comment by brandond on 24 Oct 2010 00:16 UTC

DNS has been fixed. whitestar.uoregon.edu - see e-mail for credentials.

rcubetrac commented 14 years ago

Comment by brandond on 26 Oct 2010 19:15 UTC

Changes so far are looking really great! Two comments:

The message count column in the folder management view is still using SELECT instead of STATUS:

--- program/steps/settings/manage_folders.inc.old       2010-10-26 11:18:47.000000000 -0700
+++ program/steps/settings/manage_folders.inc   2010-10-26 11:18:06.000000000 -0700
@@ -299,7 +299,7 @@
     $table->add_row(array('id' => 'rcmrow'.$idx, 'class' => join(' ', $classes)));

     $table->add('name', Q($display_folder));
-    $table->add('msgcount', ($folder[? '' : $IMAP->messagecount($folder['id']('virtual']), 'ALL', false, false)));
+    $table->add('msgcount', ($folder[? '' : $IMAP->messagecount($folder['id']('virtual']), 'ALL', false, true)));
     $table->add('subscribed', $checkbox_subscribe->show(($subscribed ? $folder_utf8 : ''),
       array('value' => $folder_utf8, 'disabled' => ($protected || $noselect) ? 'disabled' : '')));
     if ($threading_supported) {

Use of LIST (\Subscribed) is great - I'm guessing next step is to add flag to enable use of RETURN and then cache result instead of doing explicit STATUS later? This is the final thing necessary to do away with 90% of folder iteration round-trips. ie:

A0003 LIST (SUBSCRIBED) "" "*" RETURN (STATUS (MESSAGES UNSEEN))
* LIST (\Subscribed) "/" "Trash"
* STATUS "Trash" (MESSAGES 4 UNSEEN 0)
* LIST (\Subscribed) "/" "Sent"
* STATUS "Sent" (MESSAGES 6 UNSEEN 0)
* LIST (\Subscribed) "/" "Junk"
* STATUS "Junk" (MESSAGES 0 UNSEEN 0)
* LIST (\Subscribed) "/" "Drafts"
* STATUS "Drafts" (MESSAGES 0 UNSEEN 0)

Sorry if I'm bothering you, I'm just really excited to see this come together!

rcubetrac commented 14 years ago

Comment by @alecpl on 27 Oct 2010 06:55 UTC

Replying to brandond:

The message count column in the folder management view is still using SELECT instead of STATUS:[in 36ed9d7e([BR]] Fixed).

Use of LIST (\Subscribed) is great - I'm guessing next step is to add flag to enable use of RETURN and then cache result instead of doing explicit STATUS later?[[BR]] Right.

rcubetrac commented 10 years ago

Comment by @alecpl on 6 Jul 2014 09:12 UTC

Replying to brandond:

A0003 LIST (SUBSCRIBED) "" "*" RETURN (STATUS (MESSAGES UNSEEN))

This is actually not simple to implement and maybe not so useful. There are many constraints in the code that make it possible to use this in very specific cases. I'm curious what's the difference in performance between this and calling STATUS for each folder separately.

rcubetrac commented 9 years ago

Comment by @alecpl on 16 Apr 2015 10:47 UTC

The performance win is not clear and most of this ticket have been implemented. Closing.

rcubetrac commented 9 years ago

Status changed by @alecpl on 16 Apr 2015 10:47 UTC

new => closed

roundcube / roundcubemail

Support RFC5819 (LIST-STATUS) #3090