Deadlock while fetching dependencies

pbzdyl commented 3 years ago

Lein process is stuck while downloading dependencies with the following setup (I've minimized the reproduction case to that configuration):

We have the following Git projects (with links to their repos in Gitlab):

lib-a - standalone, independent project
lib-b - refers lib-a as its direct dependency
lib-c - refers lib-a and lib-b as its direct dependencies
app - refers lib-a, lib-b and lib-c as its direct dependencies

Empty ~/.m2/repository and ~/.gitlibs directories (a common situation in clean CI builds without any caches).

Steps to reproduce:

Clone the app repository

git clone https://gitlab.com/lein-git-down-test/app.git

Make sure local m2 repo ~/.m2/repository and ~/.gitlibs cache are empty:

mv ~/.m2/repository ~/.m2/repository_backup
mv ~/.gitlibs ~/.gitlibs_backup

Try to build the app project

cd app
lein jar

Output:

Retrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.pom from clojars
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.pom from central
Retrieving org/clojure/pom.contrib/0.3.0/pom.contrib-0.3.0.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit-parent/4.10.0.201712302008-r/org.eclipse.jgit-parent-4.10.0.201712302008-r.pom from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.pom from central
Retrieving org/sonatype/oss/oss-parent/6/oss-parent-6.pom from central
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom from central
Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy/0.0.9/jsch.agentproxy-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.pom from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.pom from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.pom from central
Retrieving com/jcraft/jsch/0.1.49/jsch-0.1.49.pom from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.pom from central
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.pom from clojars
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.jar from central
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.jar from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.jar from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.jar from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.jar from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.jar from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar from central
Retrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.jar from clojars
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.jar from clojars
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving lib-a/lib-a/master/lib-a-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-a/lib-a/b3140d4146072fb056cbe32ce3320ca6600e2885/pom.xml
Retrieving lib-b/lib-b/master/lib-b-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-b/lib-b/8c6e91809ea51ba8fcfdb43cc12a8a472668b8fc/pom.xml
Retrieving lib-c/lib-c/master/lib-c-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-c/lib-c/56eadc7cc0e24b883a294a6a52a9ecea908606d6/pom.xml
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Retrieving lib-a/lib-a/master/lib-a-master.jar from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.jar from gitlab
Retrieving lib-c/lib-c/master/lib-c-master.jar from gitlabRetrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.pom from clojars
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.pom from central
Retrieving org/clojure/pom.contrib/0.3.0/pom.contrib-0.3.0.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.pom from central
Retrieving org/eclipse/jgit/org.eclipse.jgit-parent/4.10.0.201712302008-r/org.eclipse.jgit-parent-4.10.0.201712302008-r.pom from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.pom from central
Retrieving org/sonatype/oss/oss-parent/6/oss-parent-6.pom from central
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom from central
Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy/0.0.9/jsch.agentproxy-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.pom from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.pom from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.pom from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.pom from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.pom from central
Retrieving com/jcraft/jsch/0.1.49/jsch-0.1.49.pom from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.pom from central
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.pom from clojars
Retrieving com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar from central
Retrieving org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.jar from central
Retrieving org/clojure/tools.gitlibs/1.0.100/tools.gitlibs-1.0.100.jar from central
Retrieving com/jcraft/jsch/0.1.54/jsch-0.1.54.jar from central
Retrieving com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.jar from central
Retrieving net/java/dev/jna/jna/4.1.0/jna-4.1.0.jar from central
Retrieving net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.jar from central
Retrieving com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.jar from central
Retrieving com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.jar from central
Retrieving com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar from central
Retrieving reifyhealth/lein-git-down/0.4.0/lein-git-down-0.4.0.jar from clojars
Retrieving lein-pprint/lein-pprint/1.3.2/lein-pprint-1.3.2.jar from clojars
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving lib-a/lib-a/master/lib-a-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-a/lib-a/b3140d4146072fb056cbe32ce3320ca6600e2885/pom.xml
Retrieving lib-b/lib-b/master/lib-b-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-b/lib-b/8c6e91809ea51ba8fcfdb43cc12a8a472668b8fc/pom.xml
Retrieving lib-c/lib-c/master/lib-c-master.pom from gitlab
Wrote /Users/piotrek/.gitlibs/libs/lib-c/lib-c/56eadc7cc0e24b883a294a6a52a9ecea908606d6/pom.xml
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Retrieving lib-a/lib-a/master/lib-a-master.jar from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.jar from gitlab
Retrieving lib-c/lib-c/master/lib-c-master.jar from gitlab

and the process is stuck with this output (there is no progress made).

Thread dump attached.

I suspect that this is related to file locking because when I deleted the ~/.m2/repository when the process was stuck, suddently it unblocked and continued with the following output:

Could not find artifact lib-a:lib-a:jar:master in central (https://repo1.maven.org/maven2/)
Could not find artifact lib-a:lib-a:jar:master in clojars (https://repo.clojars.org/)
Could not transfer artifact lib-a:lib-a:jar:master from/to gitlab (git://gitlab.com): /Users/piotrek/.m2/repository/lib-a/lib-a/master/lib-a-master.jar.part.lock (No such file or directory)
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.jar from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Created /Users/piotrek/.gitlibs/libs/lib-a/lib-a/b3140d4146072fb056cbe32ce3320ca6600e2885/target/lib-a-0.0.0-SNAPSHOT.jar
Retrieving lib-a/lib-a/master/lib-a-master.pom from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.pom from gitlab
Retrieving lib-a/lib-a/master/lib-a-master.jar from gitlab
Retrieving lib-b/lib-b/master/lib-b-master.jar from gitlab

and it got blocked again.

manderson202 commented 3 years ago

Thanks for the detailed issue description and reproduction. This issue occurs, as you show, when you have transitive dependencies also using lein-git-down. The only current workaround is to download the transitive dependencies first and have them cached, which is, of course, less than ideal. Will use this issue for status tracking.

pbzdyl commented 3 years ago

Is there a chance this issue will be fixed? (e.g. is there a potential fix that could resolve this)

manderson202 commented 3 years ago

We know the cause of the issue and we should be able to fix it, however, the fix may be a bit involved. Will need to do some more investigation to determine the full scope.

pbzdyl commented 3 years ago

Great - thank you for the information. Please, let me know if I can help (for example with testing).

reifyhealth / lein-git-down

Deadlock while fetching dependencies #46