client/string call returns nil

[canassa]

I am writing a web crawler using the http.async.client but I am having some intermittent issues when parsing the response

In my application there is a get-user-user function similar to this:

(defn get-user-page []
  (with-open [client (c/create-client :compression-enabled true)]
    (let [response (c/GET client URL)]
      (c/await response)
      (when-let [err (c/error response)]
        (do
          (println "failed processing request: " err)
          (get-user-page)))
      (java.io.StringReader. (c/string response)))))

The problem is that sometimes it raises this exception:

Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
    at clojure.lang.Util.runtimeException(Util.java:165)
    at clojure.lang.LazySeq.sval(LazySeq.java:51)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.Cons.next(Cons.java:39)
    at clojure.lang.RT.next(RT.java:580)
    at clojure.core$next.invoke(core.clj:64)
    at clojure.core$dorun.invoke(core.clj:2724)
    at legendas.core$craw.invoke(core.clj:141)
    at legendas.core$_main.invoke(core.clj:146)
    at clojure.lang.AFn.applyToHelper(AFn.java:161)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at legendas.core.main(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at clojure.core$future_call$reify__5684.deref(core.clj:6054)
    at clojure.core$deref.invoke(core.clj:2078)
    at legendas.core$cmap$step__1234$fn__1236.invoke(core.clj:123)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    ... 10 more
Caused by: java.lang.NullPointerException
    at java.io.StringReader.<init>(StringReader.java:33)
    at legendas.core$get_user_page.invoke(core.clj:65)
    at legendas.core$get_user_and_save.invoke(core.clj:106)
    at legendas.core$craw$fn__1243.invoke(core.clj:141)
    at legendas.core$cmap$fn__1229$fn__1230.invoke(core.clj:119)
    at clojure.core$binding_conveyor_fn$fn__3713.invoke(core.clj:1817)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

So I wrapped the (c/string response) in a try catch block:

(try
  (java.io.StringReader. (c/string response))
(catch Exception e
  (do
    (println "Error\n" (c/string response) (c/status response) response)
    (throw e)
    (System/exit 0))))

And this was what the println printed:

Error
nil
nil
{:id req-id__35466, :status #<core$promise$reify__5727@4fe4aec2: :pending>, :headers #<core$promise$reify__5727@2e8d6b88: :pending>, :body #<core$promise$reify__5727@468ec598: :pending>, :done #<core$promise$reify__5727@474ea258: true>, :error #<core$promise$reify__5727@7b8463fa: #<TimeoutException java.util.concurrent.TimeoutException: No response received after 60000>>}

It seems that the c/string call is returning nil and that causing the exception. What I don't get is why it happens since I am already checking for errors before calling c/string. Am I doing something wrong?

Thanks, Cesar Canassa

[neotyk]

Please check connection timeout test and request timeout tests.

Your code should be:

    (defn get-user-page []
      (with-open [client (c/create-client :compression-enabled true)]
        (let [response (c/GET client URL)]
          (c/await response)
          (if-let [err (c/error response)]
            (do
              (println "failed processing request: " err)
              (get-user-page))
            (java.io.StringReader. (c/string response))))))

[canassa]

Thank you!