Caveman V2 - EOF when POSTing body with application/json

[rayokota]

Hi,

I'm using Caveman2 on Mac OS X. I'm new to Common Lisp so I hope I'm not doing something obviously wrong. Anyway, whenever I try to submit a POST or PUT with Content-Type of application/json, I receive an END-OF-FILE from within Clack.Middleware.Json as follows:

CLACK.HANDLER:<HANDLER {1007A18513}>

Unhandled END-OF-FILE in thread #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:55384" RUNNING {1007D4DE83}>: end of file on #<FLEXI-STREAMS:FLEXI-IO-STREAM {100585E1F3}>

Backtrace for: #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:55384" RUNNING {1007D4DE83}> 0: ((LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX)) 1: (SB-IMPL::CALL-WITH-SANE-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {1006099AEB}>) 2: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {1006099ABB}>) 3: (SB-DEBUG:PRINT-BACKTRACE :STREAM #<SYNONYM-STREAM :SYMBOL SB-SYS:STDERR {1000199CB3}> :START 0 :FROM :INTERRUPTED-FRAME :COUNT NIL :PRINT-THREAD T :PRINT-FRAME-SOURCE NIL :METHOD-FRAME-STYLE NIL) 4: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<END-OF-FILE {1005F511F3}> #) 5: (SB-DEBUG::RUN-HOOK SB-EXT:INVOKE-DEBUGGER-HOOK #<END-OF-FILE {1005F511F3}>) 6: (INVOKE-DEBUGGER #<END-OF-FILE {1005F511F3}>) 7: ((:METHOD HUNCHENTOOT:MAYBE-INVOKE-DEBUGGER (T)) #<END-OF-FILE {1005F511F3}>) [fast-method] 8: (SIGNAL #<END-OF-FILE {1005F511F3}>) 9: (ERROR END-OF-FILE :STREAM #<FLEXI-STREAMS:FLEXI-IO-STREAM {100585E1F3}>) 10: (PEEK-CHAR NIL #<FLEXI-STREAMS:FLEXI-IO-STREAM {100585E1F3}> T NIL #) 11: ((:METHOD YASON::PARSE% (STREAM)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {100585E1F3}>) [fast-method] 12: (YASON:PARSE #<FLEXI-STREAMS:FLEXI-IO-STREAM {100585E1F3}> :OBJECT-KEY-FN NIL :OBJECT-AS NIL :JSON-ARRAYS-AS-VECTORS NIL :JSON-BOOLEANS-AS-SYMBOLS NIL :JSON-NULLS-AS-KEYWORD NIL) 13: ((:METHOD CLACK.COMPONENT:CALL (CLACK.MIDDLEWARE.JSON: T)) #CLACK.MIDDLEWARE.JSON:<CLACK-MIDDLEWARE-JSON {10060A3BE3}> #) [fast-method] 14: ((LAMBDA ())) 15: (SB-INT:SIMPLE-EVAL-IN-LEXENV (LET ((STANDARD-OUTPUT CLACK:CLACK-OUTPUT) (ERROR-OUTPUT CLACK:CLACK-ERROR-OUTPUT)) (CLACK.MIDDLEWARE:CALL-NEXT #CLACK.MIDDLEWARE.LET:<CLACK-MIDDLEWARE-LET {1009019A23}> (QUOTE (:BODY-PARAMETERS (:JSON #<HASH-TABLE :TEST EQUAL :COUNT 2 {1005E4BFE3}>) :REQUEST-METHOD :PUT :SCRIPT-NAME "" :PATH-INFO "/word1" :SERVER-NAME "127.0.0.1" :SERVER-PORT 8080 ...)))) #) 16: (EVAL (LET ((STANDARD-OUTPUT CLACK:CLACK-OUTPUT) (ERROR-OUTPUT CLACK:CLACK-ERROR-OUTPUT)) (CLACK.MIDDLEWARE:CALL-NEXT #CLACK.MIDDLEWARE.LET:<CLACK-MIDDLEWARE-LET {1009019A23}> (QUOTE (:BODY-PARAMETERS (:JSON #<HASH-TABLE :TEST EQUAL :COUNT 2 {1005E4BFE3}>) :REQUEST-METHOD :PUT :SCRIPT-NAME "" :PATH-INFO "/word1" :SERVER-NAME "127.0.0.1" :SERVER-PORT 8080 ...))))) 17: ((LAMBDA NIL :IN CLACK.HANDLER.HUNCHENTOOT:RUN)) 18: ((:METHOD HUNCHENTOOT:HANDLE-REQUEST (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST)) #CLACK.HANDLER.HUNCHENTOOT::<DEBUGGABLE-ACCEPTOR (host , port 8080)> #<HUNCHENTOOT:REQUEST {1008B24603}>) [fast-method] 19: ((:METHOD HUNCHENTOOT:PROCESS-REQUEST (T)) #<HUNCHENTOOT:REQUEST {1008B24603}>) [fast-method] 20: (HUNCHENTOOT::DO-WITH-ACCEPTOR-REQUEST-COUNT-INCREMENTED #CLACK.HANDLER.HUNCHENTOOT::<DEBUGGABLE-ACCEPTOR (host , port 8080)> #<CLOSURE (LAMBDA NIL :IN HUNCHENTOOT:PROCESS-CONNECTION) {10085EC5BB}>) 21: ((:METHOD HUNCHENTOOT:PROCESS-CONNECTION (HUNCHENTOOT:ACCEPTOR T)) #CLACK.HANDLER.HUNCHENTOOT::<DEBUGGABLE-ACCEPTOR (host , port 8080)> #<USOCKET:STREAM-USOCKET {1007D48C33}>) [fast-method] 22: ((:METHOD HUNCHENTOOT:PROCESS-CONNECTION (CLACK.HANDLER.HUNCHENTOOT:: T)) #CLACK.HANDLER.HUNCHENTOOT::<DEBUGGABLE-ACCEPTOR (host , port 8080)> #<USOCKET:STREAM-USOCKET {1007D48C33}>) [fast-method] 23: ((:METHOD HUNCHENTOOT:PROCESS-CONNECTION :AROUND (HUNCHENTOOT:ACCEPTOR T)) #CLACK.HANDLER.HUNCHENTOOT::<DEBUGGABLE-ACCEPTOR (host , port 8080)> #<USOCKET:STREAM-USOCKET {1007D48C33}>) [fast-method] 24: ((FLET HUNCHENTOOT::PROCESS-CONNECTION% :IN HUNCHENTOOT::HANDLE-INCOMING-CONNECTION%) #CLACK.HANDLER.HUNCHENTOOT::<DEBUGGABLE-ACCEPTOR (host , port 8080)> #<USOCKET:STREAM-USOCKET {1007D48C33}>) 25: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS)) 26: ((FLET #:WITHOUT-INTERRUPTS-BODY-1260 :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE)) 27: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE)) 28: ((FLET #:WITHOUT-INTERRUPTS-BODY-651 :IN SB-THREAD::CALL-WITH-MUTEX)) 29: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {520BCAB}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:55384" RUNNING {1007D4DE83}>> NIL T NIL) 30: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:55384" RUNNING {1007D4DE83}> #S(SB-THREAD:SEMAPHORE :NAME "Thread setup semaphore" :%COUNT 0 :WAITCOUNT 0 :MUTEX #<SB-THREAD:MUTEX (free) {1007D4DEF3}> :QUEUE #<SB-THREAD:WAITQUEUE {1007D4DF13}>) #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {1007D4DE2B}> NIL NIL NIL NIL) 31: ("foreign function: call_into_lisp") 32: ("foreign function: new_thread_trampoline") 33: ("foreign function: _pthread_start") 34: ("foreign function: thread_start")

[fukamachi]

I suppose it had fixed in 1d2889.

Quicklisp dist isn't updated until then, so use the latest Clack or wait for the April QL dist update.

[rayokota]

Thanks! I'll give it a try

[rayokota]

Using the latest Clack worked. Thanks again.

[jackcarrozzo]

What's considered the right way to temporarily use latest Clack until the quicklisp dist is out? I'm having this same issue.

[fukamachi]

What's considered the right way to temporarily use latest Clack until the quicklisp dist is out?

Try (ql:update-all-dists). This issue happened 11 months ago and I'm sure it has been fixed in the latest Quicklisp dist.

[jackcarrozzo]

Hmmm... I did indeed update-all-dists and I have clack-20150302-git in dists, but I appear to be hitting a similar issue when POSTing to a caveman2 app:

debugger invoked on a END-OF-FILE in thread
#<THREAD "hunchentoot-worker-127.0.0.1:56345" RUNNING {1007C55283}>:
  end of file on #<SB-IMPL::STRING-INPUT-STREAM {1009341CF3}>

Is there a quick way to turn on debugging/tracebacks? I am using this route:

(defroute ("/listofthings/:apikey" :method :POST) (&key apikey |things|)
  (format t "made it to here. things:~%~a~%" |things|)

  (let ((thing-list (cl-json:decode-json-from-string |things|)))
    (format t "got in: ~%~a~%" thing-list)

    (render-json
     `(:result "OK"
               :data ,thing-list))))

The output from a request is as follows, always terminating at the same point offset:

made it to here. things:
{"Brian, Dave, Jack, and Josh":{"rating":100,"occured":5},"---":{"rating":0,"occured":1},"Gretchen Parlato":{"rating":80,"occured":28},"Joe Cartwright":{"rating":0,"occured":1},"Robben Ford

Running SBCL 1.2.2.

[fukamachi]

That is definitely not the same issue to this thread's. The EOF is happened at (cl-json:decode-json-from-string |things|). It might be a bug of HTTP-Body. Could you tell me more informations the HTTP request you sent, like Content-Type and the body?

[jackcarrozzo]

Yes - here's an exact request:

POST /thingslist/1 HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 850
Accept: application/json, text/plain, */*
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://localhost:8080/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: CLACK.SESSION=1748f9e141c96e5785eb811ffcd5d414fd2120ab

artists={"Brian, Dave, Jack, and Josh":{"rating":100,"occured":5},"---":{"rating":0,"occured":1},"Gretchen Parlato":{"rating":80,"occured":28},"Joe Cartwright":{"rating":0,"occured":1},"Robben Ford & The Blue Line":{"rating":80,"occured":1},"Anders Osborne":{"rating":80,"occured":4},"Matt Schofield Trio":{"rating":80,"occured":2},"Robben Ford And The Blue Line":{"rating":80,"occured":3},"Kenny Burrell":{"rating":80,"occured":3},"Kenny Burrell and Jimmy Smith":{"rating":80,"occured":2},"Kenny Burrell and Jimmy Smith (H)":{"rating":0,"occured":1},"Kenny Burrell & John Coltrane":{"rating":80,"occured":2},"Pat Metheny":{"rating":100,"occured":27},"Matt Schofield":{"rating":100,"occured":7},"Pat Metheny & Brad Mehldau":{"rating":80,"occured":1},"Pat Metheny Group":{"rating":80,"occured":10},"Robben Ford":{"rating":100,"occured":9}}

The same result is triggered using this command:

curl -XPOST localhost:8080/thingslist/1 -d 'things={"Brian, Dave, Jack, and Josh":{"rating":100,"occured":5},"---":{"rating":0,"occured":1},"Gretchen Parlato":{"rating":80,"occured":28},"Joe Cartwright":{"rating":0,"occured":1},"Robben Ford & The Blue Line":{"rating":80,"occured":1},"Anders Osborne":{"rating":80,"occured":4},"Matt Schofield Trio":{"rating":80,"occured":2},"Robben Ford And The Blue Line":{"rating":80,"occured":3},"Kenny Burrell":{"rating":80,"occured":3},"Kenny Burrell and Jimmy Smith":{"rating":80,"occured":2},"Kenny Burrell and Jimmy Smith (H)":{"rating":0,"occured":1},"Kenny Burrell & John Coltrane":{"rating":80,"occured":2},"Pat Metheny":{"rating":100,"occured":27},"Matt Schofield":{"rating":100,"occured":7},"Pat Metheny & Brad Mehldau":{"rating":80,"occured":1},"Pat Metheny Group":{"rating":80,"occured":10},"Robben Ford":{"rating":100,"occured":9}}'

This similar but smaller structure works just fine however:

curl -XPOST localhost:8080/thingslist/1 -d 'things={"a":{"x":7,"y":8},"b":{"x":9,"y":0}}'

Any idea where I should look?

[fukamachi]

I got it. The body is not a valid urlencoded form. You have to escape special characters in percent-encoding. In your case, as & is a reserved character, it must be escaped. ", { and } would also be better to be escaped.

[fukamachi]

You might want to send it with "Content-Type: application/json".

curl -XPOST localhost:8080/thingslist/1 -H "Content-type: application/json" -d '{"things":{"Brian, Dave, Jack, and Josh":{"rating":100,"occured":5},"---":{"rating":0,"occured":1},"Gretchen Parlato":{"rating":80,"occured":28},"Joe Cartwright":{"rating":0,"occured":1},"Robben Ford & The Blue Line":{"rating":80,"occured":1},"Anders Osborne":{"rating":80,"occured":4},"Matt Schofield Trio":{"rating":80,"occured":2},"Robben Ford And The Blue Line":{"rating":80,"occured":3},"Kenny Burrell":{"rating":80,"occured":3},"Kenny Burrell and Jimmy Smith":{"rating":80,"occured":2},"Kenny Burrell and Jimmy Smith (H)":{"rating":0,"occured":1},"Kenny Burrell & John Coltrane":{"rating":80,"occured":2},"Pat Metheny":{"rating":100,"occured":27},"Matt Schofield":{"rating":100,"occured":7},"Pat Metheny & Brad Mehldau":{"rating":80,"occured":1},"Pat Metheny Group":{"rating":80,"occured":10},"Robben Ford":{"rating":100,"occured":9}}}'

[jackcarrozzo]

Ahhh awesome, thanks!