search

dakrone / clj-http

An idiomatic clojure http client wrapping the apache client. Officially supported version.
http://clojars.org/clj-http
MIT License
1.78k stars 411 forks source link

Indefinite wait in clj-http 2.x when making a HTTP call #534

Closed fhalde closed 4 years ago

fhalde commented 4 years ago

I am aware, clj-http 2.x is no longer maintained, but would appreciate if i just get a guidance around this issue as i do not have an expertise in org.apache.commons.httpclient

We've been witnessing abrupt application freeze in our test setup which makes use of clj-http.client with a reusable connection manager

Here is the thread dump that we have taken

"MultiThreadedHttpConnectionManager cleanup" #12 daemon prio=5 os_prio=0 tid=0x00002b45ec056000 nid=0x236 in Object.wait() [0x00002b45f02ff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000072a9ae070> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x000000072a9ae070> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

The conn-manager in newer version of clj-http uses PoolingHTTPClientConnectionManager and the version of clj-http I use makes use of PoolingClientConnectionManager

Is there any known bug of possible deadlock?

fhalde commented 4 years ago

I also don't understand from where is the MultiThreadedHttpConnectionManager coming in the stacktrace if the conn-manager is making use of a PoolingClientConnectionManager

fhalde commented 4 years ago

Couple of more points

We have around 1000 tests that in their test setup create a connection manager (hence 1000 connection managers) with a :timeout parameter set to 10 (seconds i believe)

We do see that we don't shutdown the connection managers. But i'm guessing the :timeout parameter does cause the cleanup to happen after the timeout has triggered

We'll definitely work to reduce the conn-manager footprint, but can you think of anything else here?

fhalde commented 4 years ago

What exactly was the default socket timeout in 2.x clj-http? was it indefinite?

dakrone commented 4 years ago

What exactly was the default socket timeout in 2.x clj-http? was it indefinite?

It appears this is the case, it waits infinitely unless otherwise specified.

fhalde commented 4 years ago

Thanks @dakrone it was indeed that case