Lein deps hangs using 100% cpu and eats more than 4 gigs of ram

[sashkachan]

I have added a maven dependency to my project. When running lein deps CPU usage spikes for the process and memory usage grows to more than 4 gigs.. Not sure what's going on under the hood, couldn't find a verbose option.

Here's my deps:

  :repositories [["nba" "http://bits.netbeans.org/nexus/content/groups/netbeans/"]]
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [org.netbeans.modules/org-netbeans-modules-php-editor "RELEASE802"]
                 [org.netbeans.modules/org-netbeans-modules-php-api-phpmodule "RELEASE802"]
                 [org.netbeans.modules/org-netbeans-modules-php-api-editor "RELEASE802"]
                 [org.netbeans.api/org-netbeans-modules-parsing-api "RELEASE802"]]

The process gets stuck at

Retrieving org/netbeans/api/org-netbeans-spi-viewmodel/RELEASE802/org-netbeans-spi-viewmodel-RELEASE802.pom from nba

Is it lein, maven or the dependency that causes the problem? On my macbook pro, I left it for around 10-15 mins max, but it'd get too hot, I always abort.

[hypirion]

It's hard to know where this origins from, but I am having an educated guess that either aether (dependency resolution lib) or lein core doesn't keep track of dependencies it has invoked. This should be verifiable by invoking aether directly and traversing the dependencies.

[glts]

Looking through old issues in this repo … after all these years this is still reproducible today. lein deps runs and then gets stuck at [org.netbeans.api/org-netbeans-spi-viewmodel].

Work-around: set :pedantic? false in the project.clj.

This has got to be a bug in Leiningen, profiling shows a lot of activity in the namespaces leiningen.core.pedantic and cemerick.pomegranate.aether. Pure Maven 3.6.0 resolves these dependencies without any issues.

[glts]

Minimal project.clj for repro with Leiningen 2.9.1:

(require 'cemerick.pomegranate.aether)

(cemerick.pomegranate.aether/register-wagon-factory!
  "http" #(org.apache.maven.wagon.providers.http.HttpWagon.))

(defproject repro "0.1.0-SNAPSHOT"
  :repositories [["nba" "http://bits.netbeans.org/nexus/content/groups/netbeans/"]]
  :dependencies [[org.netbeans.modules/org-netbeans-modules-php-editor "RELEASE802"]])

Then run lein deps: processors become very busy and the program appears to hang.

Work-arounds: set :pedantic? false; or increase the max heap size to 8GB or more (export LEIN_JVM_OPTS=-Xmx8g), dependency resolution will succeed after a few minutes.

Cause: Most of the time is spent in leiningen.core.pedantic/all-paths. The algorithm used there produces extremely large results (several million distinct paths) for very large dependency trees like that of [org.netbeans.modules/org-netbeans-modules-php-editor "RELEASE802"].

[vincent-dm]

I also ran into this issue. The problem indeed seems to be the combination of pedantic mode with multiple dependencies which rely on version ranges (in my case the software.amazon.awscdk dependencies)

It can be reproduced with this project.clj:

(defproject
  example "1.0.0"

  :dependencies [[org.clojure/clojure "1.10.1"]

                 [software.amazon.awscdk/cloudfront "1.47.1"]
                 [software.amazon.awscdk/ecs-patterns "1.47.1"]
                 [software.amazon.awscdk/efs "1.47.1"]
                 [software.amazon.awscdk/rds "1.47.1"]
                 [software.amazon.awscdk/s3 "1.47.1"]
                 [software.amazon.awscdk/ssm "1.47.1"]]

 :pedantic? :abort)

After hogging the CPU for a few minutes it prints a huuuuuge list of version conflicts (which also takes many minutes).

Maybe :pedantic? is irreconcilable with version ranges, but in that case the proper behavior would be to abort immediately and point to the version range as the culprit.

[technomancy]

I imagine this is a dupe of #2763 but the repository/deps in the repro case are no longer resolvable. If we can get an updated repro case we can investigate but without it there's not much we can do.

[vincent-dm]

When I reproduce my example above it keeps downloading tons of dependencies, but Lein doesn't hang. So for me it is fine to close it.

[technomancy]

Great; thanks.