search

nojb / ocaml-imap

Lwt-compatible IMAP4rev1 client library for OCaml
https://nojb.github.io/ocaml-imap/imap
Other
40 stars 17 forks source link

Fetching from gmail gives "bad command" #12

Closed rgrinberg closed 9 years ago

rgrinberg commented 9 years ago

I'm trying to use ocaml-imap against gmail and so far, it seems to connect, login, and search messages just fine. However, when it comes to actually fetching a message gmail seems to reject my request (shown below). I'm not that familiar with imap so I'm not sure if this is incorrect usage by me, ocaml-imap not generating the correct fetch request, or perhaps some gmail incompatibility. I think the transcript below should be helpful to diagnose that. I've also linked the full code [1] in case you want to take a glance at it.

<<< marks incoming stuff from gmail. >>> is the outgoing stuff. You can see the error from gmail being printed right before the exception.

2015-06-11 18:25:19.015546-04:00 Info <<< * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT
0 OK robot@peggedsoftware.com authenticated (Success)

2015-06-11 18:25:19.015667-04:00 Info `Ok
2015-06-11 18:25:19.015716-04:00 Info >>> 1 SELECT "[Gmail]/All Mail"

2015-06-11 18:25:19.168333-04:00 Info <<< * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Phishing $NotPhishing)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $Phishing $NotPhishing \*)] Flags permitted.
* OK [UIDVALIDITY 11] UIDs valid.
* 67 EXISTS
* 0 RECENT
* OK [UIDNEXT 68] Predicted next UID.
* OK [HIGHESTMODSEQ 3650]
1 OK [READ-WRITE] [Gmail]/All Mail selected. (Success)

2015-06-11 18:25:19.168461-04:00 Info `Ok
2015-06-11 18:25:19.168518-04:00 Info >>> 2 UID SEARCH UNSEEN

2015-06-11 18:25:19.322717-04:00 Info <<< * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 29 30 31 32 33 34 35 36 37 38 39 40 41 43 44 45 46 47 48 49 51 52 53 55 56 57 5
8 59 60 61 62 63 64 65 66 67
2 OK SEARCH completed (Success)

2015-06-11 18:25:19.322858-04:00 Info `Ok
2015-06-11 18:25:19.322935-04:00 Info Message ids: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,29,30,31,32,33,34,35,36,37,38,39,40,41,43,44,45,46,47,48,49,51,52,53,55,56,57,5
8,59,60,61,62,63,64,65,66,67
2015-06-11 18:25:19.322987-04:00 Info >>> 3 FETCH 1,2,3 RFC822.HEADER

2015-06-11 18:25:19.456999-04:00 Info <<< 3 BAD Could not parse command

2015-06-11 18:25:19.457065-04:00 Info `Error
((pid 33009) (thread_id 0)
 ((human_readable 2015-06-11T18:25:19-0400)
  (int63_ns_since_epoch 1434061519457420000))
 "unhandled exception in Async scheduler"
 ("unhandled exception"
  ((lib/monitor.ml.Error_
    ((exn
      (email/async_imap.ml.Imap_error
       "BAD: none \"Could not parse command\""))
     (backtrace
      ("Called from file \"lib/raw_deferred.ml\", line 55, characters 65-68"
       "Called from file \"lib/job_queue.ml\", line 164, characters 6-47" ""))
     (monitor
      (((name main) (here ()) (id 1) (has_seen_error true)
        (is_detached false) (kill_index 0))))))
   (Pid 33009))))

[1] Full code is here - https://gist.github.com/rgrinberg/4507557d400c5c891000

nojb commented 9 years ago

It looks like the syntax "3 FETCH 1,2,3 RFC822.HEADER" is not quite correct (IMAP servers are quite finicky about the exact syntax and Gimap in particular). Should be easy to fix. Have to run right now but will look into it later today. Thanks for reporting!

nojb commented 9 years ago

Actually I just remembered that Gimap does not allow FETCH using sequence number; I think you must use the UID FETCH command. In any case the numbers returned by UID SEARCH are unique identifiers, so doing a FETCH with those numbers would be incorrect.

I believe simply using UID FETCH instead of FETCH should solve the problem (use the ~uid:true parameter in fetch command).

rgrinberg commented 9 years ago

Unfortunately it does not fix the problem:

...
2015-06-12 01:19:13.818842-04:00 Info `Ok
2015-06-12 01:19:13.818970-04:00 Info Message ids: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,29,30,31,32,33,34,35,36,37,38,39,40,41,43,44,45,46,47,48,49,51,52,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68
2015-06-12 01:19:13.819050-04:00 Info >>> 3 UID FETCH 1,2,3 RFC822.HEADER

2015-06-12 01:19:13.951986-04:00 Info <<< 3 BAD Could not parse command
rgrinberg commented 9 years ago

I've also tried fetching multiple things at once:

2015-06-12 01:21:41.884649-04:00 Info <<< * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 29 30 31 32 33 34 35 36 37 38 39 40 41 43 44 45 46 47 48 49 51 52 53 55 56 57 58 59 60 61 62 63 64 65 66 67 68
2 OK SEARCH completed (Success)

2015-06-12 01:21:41.884781-04:00 Info `Ok
2015-06-12 01:21:41.884851-04:00 Info Message ids: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,29,30,31,32,33,34,35,36,37,38,39,40,41,43,44,45,46,47,48,49,51,52,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68
2015-06-12 01:21:41.884939-04:00 Info >>> 3 UID FETCH 1,2,3 (RFC822.HEADER BODY)

2015-06-12 01:21:42.021389-04:00 Info <<< 3 BAD Could not parse command

No cigar.

nojb commented 9 years ago

Thanks! Let me investigate a little bit further - actually all those requests are legal per the RFC, but Gimap is often tricky...

rgrinberg commented 9 years ago

FYI I just tried it with python's builtin imap lib and this is what it tries to use and it seems to work:

BGFK6 FETCH 1,2,3 (RFC822)

Looks a bit bizarre to me but perhaps it's useful to you

nojb commented 9 years ago

Ok, so I finally remembered what was particular about Gimap: you need to include the CHANGEDSINCE modifier (from the CONDSTORE extension) in your FETCH requests.

Quick review:

An IMAP server that supports the CONDSTORE extension endows each message with a timestamp (its modification sequence number or MODSEQ). The MODSEQ of the mailbox is roughly the maximum of the MODSEQs of the messages within. When a message is modified or added to a mailbox it receives a new MODSEQ which is greater than every other MODSEQ in the mailbox. These numbers can be utilised to optimise the access and caching of messages from the client side.

Just to get started you can add the ~changed:1L argument to your fetch commands and see what happens.

Let me know if this works.

PS: I am not sure what the python library is doing, if I have time I'll look into it.

nojb commented 9 years ago

Actually, scratch all the above - the problem was due to a trailing space in the command string (after the closing parenthesis ')'). Which is why adding the CHANGEDSINCE modifier was "fixing" it. I was bitten by Gimap's peculiarities before and that is why I jumped to a wrong conclusion.

Just pushed the fixed to master.

Sorry about that and thanks very much for reporting.