Artifact resolution is unreasonably slow for multi-module projects

[jwebb]

note from sbt team

This issue is now solved using cached resolution feature.

updateOptions := updateOptions.value.withConsolidatedResolution(true)

If anyone has further issues with dependency resolution performance, please provide more context and open a new Github issue.

original report

So I have a multi-module project - dozens of modules - in which many of the modules have very similar (transitive) dependencies. Each day, sbt (v0.11.2) decides it needs to re-check all those dependencies, and it does so independently for each module. So if I have, say, 20 modules depending on slf4j, sbt appears to hit the repository 20 times to re-download exactly the same pom.xml. Multiply that by a hundred-odd dependencies and the whole process is taking a couple of minutes. Seems like some caching or up-front reconciliation would be helpful here.

[charleso]

@eed3si9n My sincere apologies, that's my second dumb thing I've asked you about this week.

Works like a charm! Getting about a 35-45% speed up. Thanks so much.

[magro]

For us update time hasn't changed (~1'30), having 5 modules with different dependencies.

[fkoehler]

It is faster for us if you are in the same SBT session but not when started directly from the console. Running sbt update from the console always does:

[info] Consolidating managed dependencies to org.scala-sbt.temp#temp-resolve-7d5c97c90f4b243ad4f7331f2ce8cd4f20cbd8c6;1.0 ..

Being in a running SBT session I get

[info] Found consolidated dependency org.scala-sbt.temp#temp-resolve-7d5c97c90f4b243ad4f7331f2ce8cd4f20cbd8c6;1.0 ...

afterwards which gives me a 6 second speed up after the first run. Pretty cool.

Is that expected behaviour or should we also get that speed with a fresh SBT boot?

[jsuereth]

@fkoehler That's expected behavior. The second run we can completely skip resolution because the in-memory cache finds the previous resolved values.

Additonally, running update directly, ever, forces the FIRST cache to be ignored, which means you hit ivy. What you're seeing now is the inter-project resolution cache where we try to force ivy to share resolution graphs between things.

@eed3si9n This may violate sbt's notion of "update" should force hitting the raw ivy again, but I'm not sure.

[godenji]

So, I've tried update on 0.13.6 withConsolidatedResolution and withLatestSnapshots settings, times are roughly the same, perhaps slightly faster.

I am building Play's stable 2.3 branch from source (not 2.4 master), which is tagged as a snapshot build. Are the resolution time improvements exclusive to non-snapshot projects?

The best fix in this thread has been aggregate in update := false, at least there the dependency resolution marathon only kicks in if dep(s) have in fact changed.

[anquegi]

I also have the issue with common-logs dependency trying to use ensime with my scala, play projects, I always need to remove the ivy/cache/commons-log and then all goes wee

[jamesjieye]

It's like been waiting for days before sbt finishes resolving dependencies. This is really a wasting of time.

[dk14]

I've solved that by watching cats on youtube during resolution: https://www.youtube.com/watch?v=UIrEM_9qvZU.

[johanandren]

LOL. "Close: Solved"

[lucamolteni]

Is this really closed?

[eed3si9n]

The original case states:

So I have a multi-module project - dozens of modules - in which many of the modules have very similar (transitive) dependencies.

This particular concern is now solved using cached resolution feature. If anyone has further issues with dependency resolution performance, please provide more context.

[fommil]

I'm still seeing this being very slow in ensime-sbt and the intellij equivalent. Use ensime-server as an example (e.g. type gen-ensime). I'd expect the resolution of classifiers (sources/javadocs) to be a lot faster. This is a bottleneck for many projects enabling auto-update on their project.

[pauldraper]

Yeah, IntelliJ is still slow. But that's probably a bug on their side.

[fommil]

@pauldraper nope, I think that is calling straight out to sbt. Same as our ensime generator.

[valter-silva-au]

Still super slow a simple command like sbt update. Why is that ?

[danyaljj]

Painfully slow ...

[fkirill]

Almost 4 years passed... Look guys, are you serious about SBT at all?

[cmoro-deusto]

Does this CLOSED status mean that we should expect this to be this slow forever? Please don't get me wrong: I'm really grateful for SBT, Play Framework and the rest of the reactive platform... but guys, this is not reactive at all.

[eed3si9n]

It's closed because I've added cached resolution feature - http://www.scala-sbt.org/0.13/docs/Cached-Resolution.html

[DBassel]

+1 on SBT 0.13.11 both multi-project builds and single project

[pauldraper]

updateOptions := updateOptions.value.withCachedResolution(false)

vs

updateOptions := updateOptions.value.withCachedResolution(true)

is a night-and-day difference in multi-module projects.

I agree that resolution is still slower than necessary (particularly the first time), but that is really a whole other issue. Single-threaded resolution in a "parallel by default" build system is beyond strange, but that really can only be fixed by overhauling/replacing Ivy. And that would be enormously difficult now that SBT now needs both Ivy and Maven compatibility.

In any case, any further performance improvements should be a new issue or mailing topic.

[abrighton]

What if you need to use SNAPSHOT dependencies because you have multiple sbt projects that are not in a single multi-project build? The dependency resolution is very slow, caching won't work, and even setting offline := true does not work, since sbt (or ivy) goes online anyway...

[achieverprince]

I am new to scala, wanted to see what is it. Tried to import play-scala-forms-example, intellij is resolving dependency for last 30 min. I think something wrong with it...

[mohsenno1]

It's really annoying. I think the server firewall limits the download speed to 256 Kbps. I have 500Mbps connection but the download speed never exceed 256 kbps!!!!

Now I'm waiting about 1 hour for a new project to be initialized.

[achieverprince]

Looks like issue is open for 6 years

[mohsenno1]

I gave up learn ScalaJs. I found another alternative. Flutter from google. Let's give it a try.