dakrone / clj-http

An idiomatic clojure http client wrapping the apache client. Officially supported version.
http://clojars.org/clj-http
MIT License
1.78k stars 408 forks source link

Async requests errors #459

Open borkdude opened 6 years ago

borkdude commented 6 years ago

I'm trying to run async requests with clj-http:

(comment

  (require '[clj-http.client :as client])
  (require '[clj-http.conn-mgr :as conn])
  (require '[clj-http.core :as http-core])
  (def base "https://example.com")
  (def links (for [i (range 0 10)]
               (str base "/" i)))
  (def resps (atom []))
  (def errors (atom []))
  (time
   (client/with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
     (do
       (reset! resps [])
       (let [;; acm (conn/make-reuseable-async-conn-manager {})
             ;; ahclient (http-core/build-async-http-client {} acm base nil)
             handle-response (fn [res] (swap! resps conj res))
             handle-failure (fn [res] (swap! errors conj res))
             futures (for [l links]
                       (do
                         (debug "Getting" l)
                         (client/get l
                                     {:async? true
                                      ;; :connection-manager acm
                                      ;; :http-client ahclient
                                      }
                                     handle-response handle-failure)))]
         ;; wait for all futures to finish
         (doseq [f futures]
           (.get f))))))
  (count @resps)
  (count @errors)

  )

I keep seeing errors like:

Jul 19, 2018 9:00:55 PM org.apache.http.impl.nio.client.InternalHttpAsyncClient run
SEVERE: I/O reactor terminated abnormally
java.lang.IllegalStateException: Illegal state ACTIVE
    at org.apache.http.util.Asserts.check(Asserts.java:46)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:313)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
    at clj_http.conn_mgr.proxy$org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$ReuseableAsyncConnectionManager$fff3515b.execute(Unknown Source)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    at java.lang.Thread.run(Thread.java:745)

and

org.apache.http.ConnectionClosedException: Connection closed unexpectedly
  java.util.concurrent.ExecutionException: org.apache.http.ConnectionClosedException: Connection closed unexpectedly

Any pointers?

vemv commented 8 months ago

I'm getting the same for these simple examples, unfortunately

(client/request {:url "https://www.example.com" :method :get :async? true}
                  (fn [response]
                    (println {:OK response}))
                  (fn [ex]
                    (println {:NOK ex})))

(client/get "https://www.example.com"
              {:async? true}
              (fn [response]
                (println {:OK response}))
              (fn [ex]
                (println {:NOK ex})))

The stacktrace being:

#error {
 :cause Connection closed
 :via
 [{:type org.apache.http.ConnectionClosedException
   :message Connection closed
   :at [org.apache.http.nio.protocol.HttpAsyncRequestExecutor endOfInput HttpAsyncRequestExecutor.java 350]}]
 :trace
 [[org.apache.http.nio.protocol.HttpAsyncRequestExecutor endOfInput HttpAsyncRequestExecutor.java 350]
  [org.apache.http.impl.nio.DefaultNHttpClientConnection consumeInput DefaultNHttpClientConnection.java 261]
  [org.apache.http.impl.nio.client.InternalIODispatch onInputReady InternalIODispatch.java 81]
  [org.apache.http.impl.nio.client.InternalIODispatch onInputReady InternalIODispatch.java 39]
  [org.apache.http.impl.nio.reactor.AbstractIODispatch inputReady AbstractIODispatch.java 121]
  [org.apache.http.impl.nio.reactor.BaseIOReactor readable BaseIOReactor.java 162]
  [org.apache.http.impl.nio.reactor.AbstractIOReactor processEvent AbstractIOReactor.java 337]
  [org.apache.http.impl.nio.reactor.AbstractIOReactor processEvents AbstractIOReactor.java 315]
  [org.apache.http.impl.nio.reactor.AbstractIOReactor execute AbstractIOReactor.java 276]
  [org.apache.http.impl.nio.reactor.BaseIOReactor execute BaseIOReactor.java 104]
  [org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker run AbstractMultiworkerIOReactor.java 588]
  [java.lang.Thread run Thread.java 1583]]}}