JMAP: getMessageList filter doesn't work

[iNPUTmice]

I can't get the filters for the getMessageList command to work.

This is the command I'm trying:

[["getMessageList",{"filter":"text":"android"},"fetchThreads":true,"fetchMessages":true},"bVW9vvKJuRt3"]]

And I get an HTTP error 500 in return. Logfile has the following related entries:

Mar  3 15:45:35 localhost http[12325]: Running indexed subquery: (fuzzymatch text "ANDROID")
Mar  3 15:45:35 localhost http[12325]: match(part=(null), str="ANDROID")
Mar  3 15:45:35 localhost http[12325]: localhost.localdomain [127.0.0.1] as "webmaster@ltt.rs" with "Java/1.8.0_121"; "POST /jmap HTTP/1.0" => "HTTP/1.1 500 Internal Server Error" (error=Item does not exist)

Executing

[["getMessageList",{"filter":{},"fetchThreads":true,"fetchMessages":true},"mIyg/urBgUYi"]]

reveals that there is at least one message that has the word Android in both subject and body. Just to be sure I also tested this with searching for from=someemail@domain.tld (which should also exists)

I just rebuild cyrus from the current master just to be on the save side. The configure line I used was: XAPIAN_CONFIG="$CYRUSLIBS/bin/xapian-config-1.5" ./configure --enable-http --enable-autocreate --enable-xapian --enable-jmap --with-lmdb --prefix=/usr

My old thread #1849 should still have my config files.

[rsto]

What happens when you leave fetchThreads and fetchMessages unset, aka false?

[iNPUTmice]

What happens when you leave fetchThreads and fetchMessages unset, aka false?

The same thing happens.

Is there a way to manually query the xapian index directly? Or is that not really feasible?

[rsto]

You can check the contents of your Xapian database in cyrusconfigdir with

$ xapian-delve-1.5 cyrusconfigdir/search/c/user/cassandane/xapian
UUID = 4769ceaa-08ab-4955-a162-7abf7a2313ad
number of documents = 2
average document length = 92
document length lower bound = 39
document length upper bound = 145
highest document id ever used = 2
has positional information = true
currently open for writing = false

If you pass xapian-delve the -a flag, you'll see all indexed terms, in stemmed and and unstemmed versions and prefixed.

[iNPUTmice]

Oh this is slightly embarrassing turns out my squatter wasn't running (periodically?)

That's how my cyrus.conf looks like. How can make sure it runs regularly? Preferably when ever new email arrives.

START {
    recover       cmd="ctl_cyrusdb -r"
    squatter cmd="squatter -R"
}

SERVICES {
    imap          cmd="imapd" listen="imap" prefork=1
    imaps         cmd="imapd -s" listen="imaps" prefork=0
    lmtpunix      cmd="lmtpd -a" listen="/var/imap/socket/lmtp"
    lmtp          cmd="lmtpd -a" listen="localhost:24"
    http          cmd="httpd" listen="127.0.0.1:6000" prefork=1
}

EVENTS {
    checkpoint    cmd="ctl_cyrusdb -c" period=30
    delprune      cmd="cyr_expire -E 3" at=0400
    tlsprune      cmd="tls_prune" at=0400
}

[rsto]

You'll need to set up a sync_log channel for squatter to pick message arrivals. The man page for squatter has recently been overhauled for this.

E.g. add the following to your imapd.conf

sync_log: on
sync_log_channels: squatter

Still, a 500 error is fishy, something seems broken in your environment but I can't tell.

[iNPUTmice]

Those two lines did the trick. Thank you.

Still, a 500 error is fishy, something seems broken in your environment but I can't tell.

Well my guess is that the squatter will auto create the xapian index. And since it didn't ran it didn't create the index and the jmap search code kinda expects it to be there.

Technically this is a configuration problem but since the squatter is optional (and turned off without any configuration) the search code should probably check for the existence of the index first.

[nicolan]

Reading between the lines, does this mean that we should document the setting of those two sync_log items as mandatory in the imapd.conf for someone enabling xapian?

[brong]

@nicola, yes - we should document that you need xapian working for JMAP, and how to make xapian work :)

On Sun, 5 Mar 2017, at 14:15, Nicola N wrote:

Reading between the lines, does this mean that we should document the setting of those two sync_log items as mandatory in the imapd.conf for someone enabling xapian? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[1], or mute the thread[2].

--

Bron Gondwana

brong@fastmail.fm

Links:

1. https://github.com/cyrusimap/cyrus-imapd/issues/1854#issuecomment-284267654
2. https://github.com/notifications/unsubscribe-auth/AABE7bi7E_eiErxFAPSlQHZJSGYGjaVsks5rizQagaJpZM4MSWYj

[nicolan]

Xapian is documented as required for JMAP and how to make it work. Just checking that the sync_log lines specifically were part of Xapian pre-reqs as my understanding of squatter functionality and config is limited. (I don't think our Xapian docs mention squatter at all and our basic setup/install guide doesn't either).

So it becomes a larger question of : squatter is necessary, how does one sensibly configure it?

[elliefm]

Basically, those two settings. The squatter man page is pretty good/thorough for someone who wants to delve deeper, but that isn't helpful if one doesn't know they need it.

I think the Xapian docs should include a basic squatter configuration:

• START entry for "squatter -R ..." in cyrus.conf (or a DAEMON entry, which would be better if squatter supports it, but I don't know if it does)
• enable sync logging (i.e. sync_log: on)
• add a "squatter" sync log channel (i.e. sync_log_channels: squatter)

... and then reference the squatter man page for more info.

[elliefm]

There's also this page already, though it's pretty empty: http://www.cyrusimap.org/dev/imap/reference/admin/sop/squatter.html

[postilion]

This is exactly the sort of connect-the-dots documentation that I was suggesting to focus on during my visit. Not that I'm saying to wait, however. The sense I got from reading some of the notes coming through from beta testers, and new users/admins, is that often they don't know what they don't know, and we'll have to craft some clearer road maps for them.

@nicolan let me know if you'd like me to step into any of this. I'm pretty busy right now, but could probably shake loose some time if needed.