search

xsc / lein-ancient

Check your Projects for outdated Dependencies
MIT License
561 stars 33 forks source link

"lein ancient" calls timing out behind VPN and Proxy connection #50

Closed holgerabel closed 9 years ago

holgerabel commented 9 years ago

Hi! Today I upgraded from "lein-ancient 0.5.2" to "lein-ancient 0.6.4". I'm doing my work behind a VPN into a corporate network and need a proxy to get past the firewall. This makes connections to external "http" addresses slower.

With version 0.5.2, I could run "lein ancient ..." with this setup by just setting the "http_proxy" and "https_proxy" env variables to the right proxy values. I was able to do that today still successfully before updating to version 0.6.4. After updating to 0.6.4, I'm getting connection timeouts for every single package I have in my project's map. Below is a sample setting env variable "DEBUG=1":

Leiningen's classpath: :/usr/local/Cellar/leiningen/2.5.1/libexec/leiningen-2.5.1-standalone.jar
Applying task ancient to []
(debug) repositories: {"central" {:snapshots false, :url "https://repo1.maven.org/maven2/"}, "clojars" {:url "https://clojars.org/repo/"}}
(debug) cli options: {:dependencies? true, :plugins? false, :recursive? false, :interactive? false, :print? false, :tests? true, :profiles? true, :check-clojure? false, :qualified? false, :snapshots? false, :colors? true}
(debug) repositories: nil
(debug) -- artifact [com.microsoft/sqljdbc4 "3.0"] at [:dependencies 1] ... (2 repositories)
(debug) -- artifact [org.clojure/data.csv "0.1.2"] at [:dependencies 13] ... (2 repositories)
(debug) -- artifact [org.clojure/data.json "0.2.6"] at [:dependencies 12] ... (2 repositories)
(debug) -- artifact [org.jfree/jfreechart "1.0.19"] at [:dependencies 11] ... (2 repositories)
(debug) -- artifact [incanter/incanter-charts "1.9.0"] at [:dependencies 10] ... (2 repositories)
(debug) -- artifact [incanter/incanter-io "1.9.0"] at [:dependencies 9] ... (2 repositories)
(debug) -- artifact [incanter/incanter-core "1.9.0"] at [:dependencies 8] ... (2 repositories)
(debug) -- artifact [org.slf4j/slf4j-log4j12 "1.7.10"] at [:dependencies 7] ... (2 repositories)
(debug) -- artifact [org.clojure/tools.logging "0.3.1"] at [:dependencies 6] ... (2 repositories)
(debug) -- artifact [net.mikera/core.matrix "0.33.2"] at [:dependencies 5] ... (2 repositories)
(debug) -- artifact [log4j "1.2.17"] at [:dependencies 4] ... (2 repositories)
(debug) -- artifact [org.clojure/java.jdbc "0.3.6"] at [:dependencies 3] ... (2 repositories)
(debug) -- artifact [net.sourceforge.jtds/jtds "1.3.1"] at [:dependencies 2] ... (2 repositories)
(warn)  [clojars] - Δ 6334ms - failure when checking incanter/incanter-io: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6336ms - failure when checking org.slf4j/slf4j-log4j12: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6333ms - failure when checking incanter/incanter-core: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6337ms - failure when checking com.microsoft/sqljdbc4: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6334ms - failure when checking org.clojure/tools.logging: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6334ms - failure when checking org.clojure/java.jdbc: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6335ms - failure when checking org.clojure/data.csv: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6335ms - failure when checking org.clojure/data.json: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6336ms - failure when checking log4j/log4j: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6335ms - failure when checking incanter/incanter-charts: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6335ms - failure when checking org.jfree/jfreechart: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6332ms - failure when checking net.mikera/core.matrix: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [clojars] - Δ 6334ms - failure when checking net.sourceforge.jtds/jtds: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8341ms - failure when checking org.clojure/java.jdbc: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8344ms - failure when checking incanter/incanter-io: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8343ms - failure when checking org.jfree/jfreechart: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8340ms - failure when checking net.mikera/core.matrix: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8341ms - failure when checking net.sourceforge.jtds/jtds: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8349ms - failure when checking com.microsoft/sqljdbc4: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8345ms - failure when checking log4j/log4j: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8346ms - failure when checking org.slf4j/slf4j-log4j12: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8343ms - failure when checking incanter/incanter-charts: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8341ms - failure when checking incanter/incanter-core: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8343ms - failure when checking org.clojure/data.csv: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8342ms - failure when checking org.clojure/tools.logging: org.httpkit.client.TimeoutException: connect timeout: 3000ms
(warn)  [central] - Δ 8344ms - failure when checking org.clojure/data.json: org.httpkit.client.TimeoutException: connect timeout: 3000ms
all artifacts are up-to-date.

Is there a way to configure to timeout to be longer than "3000ms" so that I can run "lein ancient" in this setup. I can still work by disconnecting from the VPN and use my direct internet connection, but for work purposes this is cumbersome.

Thank you so much! Best regards, Holger

xsc commented 9 years ago

@holgerabel The latest SNAPSHOT offers a way to configure the timeout using a http_timeout environment variable. Setting it to 0 will cause a timeout of an hour, any other value represents the timeout in milliseconds. Can you check whether this works for you?

(Edit: The timeout is also automatically set to 30s if http_proxy or https_proxy is detected in the environment.)

$ http_timeout=0 lein ancient

Artifact Coordinates

[lein-ancient "0.6.5-SNAPSHOT"]
holgerabel commented 9 years ago

Yannick, Thank for getting back to me on a weekend. I tried it out and the timeout is being pushed out as you indicate, but the connection isn't ultimately made. I tried "http_timeout=0", "http_timeout=10000", etc.. I wanted to give you a new lead I have, and it is pointing to the "https_proxy" env variable not being processed correctly by lein-ancient. To get to this conclusion, I reverted back to "0.5.2" and set env "DEBUG=1" to see the what happened with "https_proxy" correctly set to my company's "https" server and then unsetting it. Here is the behavior I got from 0.5.2:

export https_proxy=http://<my company's https proxy server>
Leiningen's classpath: :/usr/local/Cellar/leiningen/2.5.1/libexec/leiningen-2.5.1-standalone.jar
Applying task ancient to []
(verbose) Checking 8 Dependencies using 2 Repositories ...
(verbose) Checking clj-http/clj-http (current version: "1.0.1" at [:dependencies 1]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/clj-http/clj-http/maven-metadata.xml ...
(verbose)   Trying to retrieve https://clojars.org/repo/clj-http/clj-http/maven-metadata.xml ...
(verbose)   Got 2379 byte(s) of data.
(verbose) Checking net.mikera/core.matrix (current version: "0.33.2" at [:dependencies 2]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/net/mikera/core.matrix/maven-metadata.xml ...
(verbose)   Trying to retrieve https://clojars.org/repo/net/mikera/core.matrix/maven-metadata.xml ...
(verbose)   Got 1826 byte(s) of data.
(verbose) Checking incanter/incanter-core (current version: "1.9.0" at [:dependencies 3]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/incanter/incanter-core/maven-metadata.xml ...
(verbose)   Trying to retrieve https://clojars.org/repo/incanter/incanter-core/maven-metadata.xml ...
(verbose)   Got 935 byte(s) of data.
(verbose) Checking incanter/incanter-io (current version: "1.9.0" at [:dependencies 4]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/incanter/incanter-io/maven-metadata.xml ...
(verbose)   Trying to retrieve https://clojars.org/repo/incanter/incanter-io/maven-metadata.xml ...
(verbose)   Got 933 byte(s) of data.
(verbose) Checking incanter/incanter-charts (current version: "1.9.0" at [:dependencies 5]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/incanter/incanter-charts/maven-metadata.xml ...
(verbose)   Trying to retrieve https://clojars.org/repo/incanter/incanter-charts/maven-metadata.xml ...
(verbose)   Got 937 byte(s) of data.
(verbose) Checking org.clojure/data.json (current version: "0.2.6" at [:dependencies 6]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/org/clojure/data.json/maven-metadata.xml ...
(verbose)   Got 637 byte(s) of data.
(verbose) Checking org.clojure/data.csv (current version: "0.1.2" at [:dependencies 7]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/org/clojure/data.csv/maven-metadata.xml ...
(verbose)   Got 388 byte(s) of data.
(verbose) Checking com.taoensso/timbre (current version: "3.4.0" at [:dependencies 8]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/com/taoensso/timbre/maven-metadata.xml ...
(verbose)   Trying to retrieve https://clojars.org/repo/com/taoensso/timbre/maven-metadata.xml ...
(verbose)   Got 2365 byte(s) of data.
(verbose) Done.
export https_proxy=
Leiningen's classpath: :/usr/local/Cellar/leiningen/2.5.1/libexec/leiningen-2.5.1-standalone.jar
Applying task ancient to []
(verbose) Checking 8 Dependencies using 2 Repositories ...
(verbose) Checking clj-http/clj-http (current version: "1.0.1" at [:dependencies 1]) ...
(verbose)   Trying to retrieve https://repo1.maven.org/maven2/clj-http/clj-http/maven-metadata.xml ...
<it hangs after this line>

So basically it seems like while in 0.5.2 the "https_proxy" env variable was being understood correctly, in "0.6.4" or "0.6.5-SNAPSHOT" this is no longer the case, as the similar behavior I get in 0.5.2 by unsetting "https_proxy" is what "0.6.4" and "0.6.5-SNAPSHOT" show regardless of the setting or unsetting "https_proxy".

I wanted to help so I'm going to try all version >0.5.2 to <0.6.4 to see if I can find where this behavior first shows up and I'll add this information as a new comment.

Thank you again for quickly spinning a new version and hopefully the information gives you additional hints on what can be going on.

Thanks again, Holger

holgerabel commented 9 years ago

Hi Yannick The issue of misunderstanding the "https_proxy" env variable seems to have been triggered in the transition from 0.5.5 to 0.6.0. 0.5.5 still works ok by setting "https_proxy", but on 0.6.0 the timeout already happens, and so does for the newer versions. Hopefully this helps some more.

Best regards, Holger

xsc commented 9 years ago

@holgerabel Thanks for looking into this. It seems that http-kit (the library used for HTTP I/O) does not support proxies. Seems like it has to be replaced (e.g. by clj-http) or patched.

I'll let you know if there's some movement.

xsc commented 9 years ago

@holgerabel The SNAPSHOT has been updated and initial experiments are promising. Could you try again? (Maybe you have to call lein -U deps to update the SNAPSHOT.)

holgerabel commented 9 years ago

@xsc You did it!! It works perfectly! I owe you a beer for working on this so quickly :). Thank you again!

xsc commented 9 years ago

@holgerabel No problem, I'm glad the issues are resolved. :)

Released as

[lein-ancient "0.6.5"]