Long response got cut off when `wrap-base`

[cmal]

The url get response data cut: http://localhost:3000/www/stock/klinedata?stockid=300370.SZ&period=D

the original url: https://www.joudou.com/stockinfogate/stock/klinedata?stockid=300370.SZ&period=D

The error message are:

2017-05-20 00:19:02,287 [qtp1106269094-16] WARN  org.eclipse.jetty.server.HttpChannel - //localhost:3000/www/stock/klinedata?stockid=300370.SZ&period=D
clojure.lang.ExceptionInfo: Couldnt' write to stream
    at clojure.core$ex_info.invokeStatic(core.clj:4617)
    at clojure.core$ex_info.invoke(core.clj:4617)
    at qbits.jet.servlet$write_stream_BANG_.invokeStatic(servlet.clj:131)
    at qbits.jet.servlet$write_stream_BANG_.invoke(servlet.clj:126)
    at qbits.jet.servlet$eval37883$fn__37884.invoke(servlet.clj:153)
    at qbits.jet.servlet$eval37796$fn__37797$G__37787__37806.invoke(servlet.clj:88)
    at qbits.jet.servlet$set_response_body_BANG_.invokeStatic(servlet.clj:93)
    at qbits.jet.servlet$set_response_body_BANG_.invoke(servlet.clj:91)
    at qbits.jet.servlet$set_body_BANG_.invokeStatic(servlet.clj:229)
    at qbits.jet.servlet$set_body_BANG_.invoke(servlet.clj:220)
    at qbits.jet.servlet$eval38053$fn__38054.invoke(servlet.clj:263)
    at qbits.jet.servlet$eval38024$fn__38025$G__38015__38032.invoke(servlet.clj:236)
    at qbits.jet.servlet$update_response.invokeStatic(servlet.clj:241)
    at qbits.jet.servlet$update_response.invoke(servlet.clj:239)
    at qbits.jet.server$make_handler$fn__38414.invoke(server.clj:78)
    at qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:518)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
    at java.lang.Thread.run(Thread.java:745)

I use the following handler:

(def app-routes
  (routes
   (-> #'home-routes
       (wrap-routes middleware/wrap-csrf)
       (wrap-routes middleware/wrap-formats))
   (middleware/wrap-proxy
    (route/not-found
     (:body
      (error-page {:status 404
                   :title "page not found"}))))))

(defn app [] (middleware/wrap-base #'app-routes))

I add the wrap-proxy to get response from another server.

I tried to get the raw req and client/request the req, then I got the full response in REPL. So I am sure that the response got by the function returned by wrap-proxy is the complete. Is this because of something in the wrap-base who cause the response to be cut off?

Thanks.

[cmal]

The related codes are here:

(defn- build-url [host path query-string]
  (let [url (str host path)]
    (if (not-empty query-string)
      (str url "?" query-string)
      url)))

(defn prepare-cookies
  "Removes the :domain and :secure keys and converts the :expires key (a Date)
  to a string in the ring response map resp. Returns resp with cookies properly
  munged."
  [resp]
  (let [prepare #(-> (update-in % [1 :expires] str)
                     (update-in [1] dissoc :domain :secure))]
    (assoc resp :cookies (into {} (map prepare (:cookies resp))))))

(defn slurp-binary
  "Reads len bytes from InputStream is and returns a byte array."
  [^java.io.InputStream is len]
  (when (and is
             (> len 0))
    (with-open [rdr is]
      (let [buf (byte-array len)]
        (.read rdr buf)
        buf))))

(defn create-proxy [handler fns]
  (let [identifier-fn (get fns :identifier-fn identity)
        host-fn (get fns :host-fn {})
        path-fn (get fns :path-fn identity)]
    (fn [request]
      (let [request-key (identifier-fn request)
            host (host-fn request-key)
            stripped-headers (dissoc (:headers request) "content-length" "host")
            path (path-fn (:uri request))
            ]
        (wrap-cookies
         (if host
           (->
            {:url              (build-url host (path-fn (:uri request)) (:query-string request))
             :method           (:request-method request)
             :body             (if-let [len (get-in request [:headers "content-length"])]
                                 (slurp-binary
                                  (:body request)
                                  (Integer/parseInt len)))
             :headers          stripped-headers
             :follow-redirects true
             :throw-exceptions false
             :as               :stream
             :insecure?        true
             }
            client/request
            (update-in [:headers] dissoc "Transfer-Encoding")
            prepare-cookies)
           (handler request)))))))

(defn wrap-proxy [handler]
  (-> handler
      (create-proxy
       {:identifier-fn :uri
        :host-fn (fn [^String uri]
                   (cond
                     ;; (.startsWith uri "/www") "http://localhost:6787/p/www/stockinfogate"
                     (.startsWith uri "/www") "https://m.joudou.com/p/www/stockinfogate"
                     (.startsWith uri "/beta") "https://m.joudou.com/p/beta/stockinfogate"
                     :else nil))
        :path-fn (fn [^String uri]
                   (subs uri (inc (.indexOf (rest uri) \/)))
                   #_(s/join "/" (cons "" (drop 2 (s/split uri #"/")))))})))

[yogthos]

I can't think of anything that should cut off the response in the middleware off top of my head. However, Jetty might have a default response size limit. One thing to try would be to swap the http server to immutant or http-kit if you're not relying on any jetty specific functionality.

[yogthos]

Immutant docs have an example of creating chunked responses under the HTTP Streams section http://immutant.org/documentation/current/apidoc/guide-web.html

[cmal]

@yogthos Ahh.. Thank you for you reply. The server of the original url is developed by my team, too. It is also served by jetty. Is it possible that the jetty of my original server can serve the large response (maybe 500KB) while the Luminus jetty cannot?

[yogthos]

You might have to tune it to server larger responses. Take a look here for the available options. Increasing the buffer size might do the trick.

[cmal]

I found that sometimes it served one chunk (64kb) and sometimes two. Maybe always the last one chunk will get lost. I'll try and comment here later. Thank you for your kind.

[yogthos]

No problem, hopefully this does the trick.

[cmal]

I tried to tune jetty but nothing changed. Then I changed to http-kit but the problem is just the same. The response get cut off at the same place. But after I try to modify (to add log) and compile the wrap-proxy in cider, it works. I wonder why. So now I have http-kit and a working server with proxy. :smile:

[yogthos]

Hmm one thing I just noticed is you're doing wrap-cookies inside the request handler fn in create-proxy, but if it's the default wrap-cookies middleware function, it expects the handler as its input as opposed to the request:

(defn create-proxy [handler fns]
  (let [identifier-fn (get fns :identifier-fn identity)
        host-fn (get fns :host-fn {})
        path-fn (get fns :path-fn identity)]
    (fn [request]
      (let [request-key (identifier-fn request)
            host (host-fn request-key)
            stripped-headers (dissoc (:headers request) "content-length" "host")
            path (path-fn (:uri request))
            ]
        (wrap-cookies
          ...)))))

This might be causing the odd behavior. Internally, wrap-cookies does:

(fn
     ([request]
      (-> request
          (cookies-request options)
          handler
          (cookies-response options)))

So you probably want:

(defn create-proxy [handler fns]
  (let [identifier-fn (get fns :identifier-fn identity)
        host-fn (get fns :host-fn {})
        path-fn (get fns :path-fn identity)]
    (fn [request]
      (let [request-key (identifier-fn request)
            host (host-fn request-key)
            stripped-headers (dissoc (:headers request) "content-length" "host")
            path (path-fn (:uri request))
            ]
        (-> (if host ... )
              (cookies-request options)
              handler
              (cookies-response options))))))

[cmal]

Ahh.. Thank you very much!