Support CONDSTORE

[manishrjain]

Often, multiple IMAP (RFC 3501) clients need to coordinate changes to a common IMAP mailbox. Examples include different clients working on behalf of the same user, and multiple users accessing shared mailboxes. These clients need a mechanism to synchronize state changes for messages within the mailbox. They must be able to guarantee that only one client can change message state (e.g., message flags) at any time. An example of such an application is use of an IMAP mailbox as a message queue with multiple dequeueing clients.

These days many clients work with the same mail server, your desktop, laptop, mobile devices are all retrieving mails and updating them. So, it would be safe to support this IMAP extension which can be used to avoid overwriting some other client's changes.

https://tools.ietf.org/html/rfc4551

[emersion]

RFC 4551 has been obsoleted by https://tools.ietf.org/html/rfc7162

[andris9]

RFC 4551 might be obsoleted but major IMAP servers like Gmail still go with it, there aren't too many servers supporting RFC 7162 QRESYNC. This is also why I decided to go with CONDSTORE as defined in RFC 4551 in my own IMAP server called Wild Duck (though it's not built in Go but on Node.js+MongoDB)

[emersion]

CONDSTORE hasn't been obsoleted as it's (re-)defined (alongside QRESYNC) in RFC 7162. Are there any major differences between RFC 4551 CONDSTORE and RFC 7162 CONDSTORE?

[andris9]

There are only minor differences between RFC4551 and RFC7162 CONDSTORE specifications like 64 vs 63 bit numbers for MODSEQ, so in general these two are the same. I went with RFC 4551 in Wild Duck so I would not have to implement QRESYNC, otherwise I could not say that I support this RFC.

In short, RFC7162 only makes sense over RFC4551 if you want to use QRESYNC. If you do not use QRESYNC, then RFC4551 is probably more accurate RFC to list as supported.

[emersion]

Okay, thanks for the details!

[andris9]

Btw, there's nothing wrong with QRESYNC. I only skipped it because Gmail/Hotmail/Yahoo/AOL etc. do not support it and most new IMAP client applications optimize for these larger providers first. So there's not much point implementing such a complex feature if no-one uses it. CONDSTORE is actually not so common either in large providers but Gmail supports it, so clients usually support it as well.

[sapiens-sapide]

issue closed because it's not WIP ?
@emersion : I need to implement RFC7162 client side. Any advise on where to start from ? Thanks.

[foxcpp]

go-imap extensions system is a kind of mess. It is hard to extend existing commands, I think this we might want to reconsider how it should work, taking into account the number of extensions that modify existing commands instead of adding new.

[emersion]

issue closed because it's not WIP ?

No, issue isn't closed.

go-imap extensions system is a kind of mess.

Agree. Do you think we should implement everything in go-imap itself?

[foxcpp]

Agree. Do you think we should implement everything in go-imap itself?

Not sure. I'm afraid this will require all go-imap server backends to support all extensions. Making it harder to implement at least minimal backend.

I think all we need is to add more "extension points". Like, hooks to parse & format new response codes. Something similar to opaque FetchItems and StatusItems maps in Message and Status objects. Etcetera. This is a big complexity increase, though...

[foxcpp]

Not sure. I'm afraid this will require all go-imap server backends to support all extensions. Making it harder to implement at least minimal backend.

On the other hand... Probably it is not a bad thing either, sane IMAP4 server should definitely support some extensions like UIDPLUS and CONDSTORE.

[emersion]

Not sure. I'm afraid this will require all go-imap server backends to support all extensions. Making it harder to implement at least minimal backend.

Well just like BackendUpdater we can enable features if the backend supports them.

[emersion]

The main issue is that IMAP isn't modular at all. Many commands redefine how other things work and basic assumptions from RFC3501 aren't true anymore. So trying to stick to a modular design like go-imap's is kind of difficult.

[foxcpp]

Here is a challenge then: UIDPLUS and CONDSTORE extensions are on. How go-imap should interact with backend to make use of them (interfaces?)

How backend interface should look without these extensions? (don't answer, we know it) How backend interface should look with UIDPLUS only? (can figure out, ok) How backend interface should look with CONDSTORE only? (can figure out, ok) How backend interface should look when both extensions are supported? Side note: Interface should allow making use of both extensions at the same time.

[foxcpp]

I guess backend methods like CreateMessage need support for arbitrary opaque inputs and outputs. Like, response with COPYUID information, etc. As I said:

Something similar to opaque FetchItems and StatusItems maps in Message and Status objects.

[emersion]

How backend interface should look when both extensions are supported?

I believe the answer to this question can only be harder if we have a modular design.

[foxcpp]

Yeah, can agree, so I guess it is time to get all these extensions into go-imap.

I guess backend methods like CreateMessage need support for arbitrary opaque inputs and outputs.

A similar problem is with client interface though. But it can be handled like "just add arguments for everything". Tho we might want to put these arguments into a struct since many of them are optional.

[emersion]

TBH if you think we can keep a modular design while supporting all extensions, I'm all for it. If having lots of repos to keep in sync is an issue, we could have a imap/x package for "officially supported" extensions.

[sapiens-sapide]

I can spend few hours on RFC7162 client side, if you need help tell me what to do.

[kompiuter]

@sapiens-sapide I'm also interested in this extension, have you done any work on this?

[andris9]

In WildDuck IMAP server, where I had similar hurdles, I did not add CONDSTORE as a separate extension but modified all commands that are affected to support it (SELECT, EXAMINE, STORE, UID STORE, FETCH, UID FETCH, SEARCH).

For example SELECT looks for CONDSTORE argument, FETCH checks for CHANGEDSINCE extension and acts accordingly, etc. Doing this everything by appending existing handlers via plugins is probably quite difficult.

[sapiens-sapide]

@kompiuter : for now I hacked go-imap response handlers with NewDebugWriter to parse server responses with a regex and extract relevant response code… quick and dirty ! ;-)

[emersion]

Oh derp, please don't!

We have plans to merge all extensions into go-imap itself. There is a dev branch that you can send pull requests to for incompatible changes.

[emersion]

Client support is now merged.