can I skip re-fetching of the repos?

[SethTisue]

now that the community build has over 160 projects in it, when I'm testing changes locally, doing git fetch on all the repos to see whether each project's dependencies need to be re-extracted takes a while.

this is by far the biggest factor causing a slow feedback loop on typical changes.

is there a way to tell dbuild "I don't care about any remote changes right now, just use whatever you already have locally"?

[cunei]

I checked a recent community build run, containing 177 projects. The time spent re-fetching the various projects from their GitHub repo is exactly 56.1 seconds, in total. So you would not save much in the end, I am afraid. Here's the full list:

[acyclic] Took: 00h 00m 00.4s
[akka-actor] Took: 00h 00m 01.7s
[akka-contrib-extra] Took: 00h 00m 00.2s
[akka-http] Took: 00h 00m 00.4s
[akka-http-cors] Took: 00h 00m 00.2s
[akka-http-json] Took: 00h 00m 00.3s
[akka-http-session] Took: 00h 00m 00.3s
[akka-more] Took: 00h 00m 01.6s
[akka-persistence-cassandra] Took: 00h 00m 00.3s
[akka-persistence-jdbc] Took: 00h 00m 00.3s
[algebra] Took: 00h 00m 00.2s
[ammonite] Took: 00h 00m 00.3s
[ammonite-ops] Took: 00h 00m 00.4s
[argonaut] Took: 00h 00m 00.4s
[argonaut-shapeless] Took: 00h 00m 00.3s
[atto] Took: 00h 00m 00.3s
[autowire] Took: 00h 00m 00.3s
[base64] Took: 00h 00m 00.2s
[better-files] Took: 00h 00m 00.2s
[blaze] Took: 00h 00m 00.3s
[breeze] Took: 00h 00m 00.3s
[cachecontrol] Took: 00h 00m 00.4s
[case-app] Took: 00h 00m 00.4s
[catalysts] Took: 00h 00m 00.3s
[cats] Took: 00h 00m 00.5s
[cats-effect] Took: 00h 00m 00.3s
[circe] Took: 00h 00m 00.4s
[circe-config] Took: 00h 00m 00.2s
[classutil] Took: 00h 00m 00.2s
[cloc-plugin] Took: 00h 00m 00.2s
[conductr-lib] Took: 00h 00m 00.3s
[coursier] Took: 00h 00m 00.4s
[curryhoward] Took: 00h 00m 00.3s
[discipline] Took: 00h 00m 00.2s
[dispatch] Took: 00h 00m 00.4s
[doodle] Took: 00h 00m 00.3s
[eff] Took: 00h 00m 00.2s
[elastic4s] Took: 00h 00m 00.3s
[fansi] Took: 00h 00m 00.3s
[fast-string-interpolator] Took: 00h 00m 00.3s
[fastparse] Took: 00h 00m 00.3s
[fs2] Took: 00h 00m 00.6s
[fs2-reactive-streams] Took: 00h 00m 00.3s
[genjavadoc] Took: 00h 00m 00.4s
[geny] Took: 00h 00m 00.3s
[gigahorse] Took: 00h 00m 00.2s
[giter8] Took: 00h 00m 00.3s
[github4s] Took: 00h 00m 00.2s
[grizzled] Took: 00h 00m 00.2s
[http4s] Took: 00h 00m 00.5s
[http4s-parboiled2] Took: 00h 00m 00.4s
[http4s-websocket] Took: 00h 00m 00.3s
[jackson-module-scala] Took: 00h 00m 00.4s
[jawn-0-10] Took: 00h 00m 00.4s
[jawn-0-11] Took: 00h 00m 00.4s
[jawn-fs2] Took: 00h 00m 00.3s
[json4s] Took: 00h 00m 00.4s
[kind-projector] Took: 00h 00m 00.4s
[kxbmap-configs] Took: 00h 00m 00.2s
[lagom] Took: 00h 00m 00.5s
[lift-json] Took: 00h 00m 00.4s
[lightbend-emoji] Took: 00h 00m 00.2s
[linter] Took: 00h 00m 00.2s
[log4s] Took: 00h 00m 00.2s
[machinist] Took: 00h 00m 00.2s
[macro-compat] Took: 00h 00m 00.3s
[macro-paradise] Took: 00h 00m 00.2s
[magnolia] Took: 00h 00m 00.3s
[meta-paradise] Took: 00h 00m 00.2s
[metaconfig-new] Took: 00h 00m 00.2s
[metaconfig-old] Took: 00h 00m 00.3s
[metrics-scala] Took: 00h 00m 00.2s
[mima] Took: 00h 00m 00.2s
[minitest] Took: 00h 00m 00.3s
[monix] Took: 00h 00m 00.3s
[monocle] Took: 00h 00m 00.3s
[mouse] Took: 00h 00m 00.3s
[multibot] Took: 00h 00m 00.2s
[nscala-time] Took: 00h 00m 00.3s
[nyaya] Took: 00h 00m 00.3s
[paiges] Took: 00h 00m 00.3s
[paradox] Took: 00h 00m 00.4s
[parboiled] Took: 00h 00m 00.4s
[parboiled2] Took: 00h 00m 00.4s
[pascal] Took: 00h 00m 00.3s
[pcplod] Took: 00h 00m 00.3s
[play-core] Took: 00h 00m 01.4s
[play-doc] Took: 00h 00m 00.4s
[play-json] Took: 00h 00m 00.4s
[play-webgoat] Took: 00h 00m 00.3s
[play-ws] Took: 00h 00m 00.5s
[pprint] Took: 00h 00m 00.2s
[pureconfig] Took: 00h 00m 00.3s
[refined] Took: 00h 00m 00.5s
[sbinary] Took: 00h 00m 00.2s
[sbt] Took: 00h 00m 00.3s
[sbt-io] Took: 00h 00m 00.2s
[sbt-librarymanagement] Took: 00h 00m 00.2s
[sbt-testng] Took: 00h 00m 00.3s
[sbt-util] Took: 00h 00m 00.2s
[scala-xml] Took: 00h 00m 00.3s
[scala-async] Took: 00h 00m 00.4s
[scala-collection-compat] Took: 00h 00m 00.2s
[scala-collections-laws] Took: 00h 00m 00.3s
[scala-continuations] Took: 00h 00m 00.2s
[scala-debugger] Took: 00h 00m 00.2s
[scala-gopher] Took: 00h 00m 00.2s
[scala-java8-compat] Took: 00h 00m 00.3s
[scala-js] Took: 00h 00m 00.5s
[scala-js-stubs] Took: 00h 00m 00.5s
[scala-logging] Took: 00h 00m 00.2s
[scala-newtype] Took: 00h 00m 00.2s
[scala-parser-combinators] Took: 00h 00m 00.3s
[scala-partest] Took: 00h 00m 00.3s
[scala-records] Took: 00h 00m 00.3s
[scala-refactoring] Took: 00h 00m 00.2s
[scala-sculpt] Took: 00h 00m 00.2s
[scala-ssh] Took: 00h 00m 00.2s
[scala-stm] Took: 00h 00m 00.2s
[scala-swing] Took: 00h 00m 00.2s
[scala-xml-quote] Took: 00h 00m 00.2s
[scalacheck] Took: 00h 00m 00.4s
[scalacheck-shapeless] Took: 00h 00m 00.2s
[scalachess] Took: 00h 00m 00.2s
[scaladex] Took: 00h 00m 00.2s
[scalafix] Took: 00h 00m 00.2s
[scalafmt] Took: 00h 00m 00.2s
[scalaj-http] Took: 00h 00m 00.2s
[scalajson] Took: 00h 00m 00.1s
[scalalib] Took: 00h 00m 00.2s
[scalameta-1] Took: 00h 00m 00.2s
[scalameta-2] Took: 00h 00m 00.2s
[scalameter] Took: 00h 00m 00.2s
[scalamock] Took: 00h 00m 00.2s
[scalapb] Took: 00h 00m 00.2s
[scalapb-lenses] Took: 00h 00m 00.2s
[scalaprops] Took: 00h 00m 00.2s
[scalariform] Took: 00h 00m 00.2s
[scalasti] Took: 00h 00m 00.2s
[scalastyle] Took: 00h 00m 00.2s
[scalatags] Took: 00h 00m 00.3s
[scalatest] Took: 00h 00m 00.3s
[scalatest-tests] Took: 00h 00m 00.3s
[scalatex] Took: 00h 00m 00.3s
[scalaz] Took: 00h 00m 00.6s
[scalaz8] Took: 00h 00m 00.4s
[scalikejdbc] Took: 00h 00m 00.3s
[scallop] Took: 00h 00m 00.2s
[scapegoat] Took: 00h 00m 00.3s
[scodec] Took: 00h 00m 00.2s
[scodec-bits] Took: 00h 00m 00.3s
[scopt] Took: 00h 00m 00.2s
[scoverage] Took: 00h 00m 00.2s
[semanticdb-sbt] Took: 00h 00m 00.2s
[shapeless] Took: 00h 00m 00.3s
[silencer] Took: 00h 00m 00.3s
[simulacrum] Took: 00h 00m 00.3s
[singleton-ops] Took: 00h 00m 00.3s
[sjson-new] Took: 00h 00m 00.4s
[sksamuel-exts] Took: 00h 00m 00.2s
[slick] Took: 00h 00m 00.3s
[sourcecode] Took: 00h 00m 00.2s
[specs2] Took: 00h 00m 00.9s
[specs2-scalaz] Took: 00h 00m 00.2s
[spire] Took: 00h 00m 00.8s
[spray-json] Took: 00h 00m 00.3s
[ssl-config] Took: 00h 00m 00.2s
[sttp] Took: 00h 00m 00.2s
[tut] Took: 00h 00m 00.3s
[twirl] Took: 00h 00m 00.3s
[twitter-util] Took: 00h 00m 00.4s
[twotails] Took: 00h 00m 00.2s
[unfiltered] Took: 00h 00m 00.5s
[upickle] Took: 00h 00m 00.2s
[utest] Took: 00h 00m 00.2s
[wartremover] Took: 00h 00m 00.4s
[zinc] Took: 00h 00m 00.7s

[SethTisue]

@cunei we discussed this in person in Montreal last week and I think I was able to convince you that a "don't go to GitHub, just use what you have" flag would be useful to me in community build work. and this is the reminder you asked for :-)

[SethTisue]

I feel like we discussed a few other related possibilities, but I'm forgetting now what they were.

I think one of them might have been, can we not only skip talking to GitHub, but also skip recomputing the inter-project dependencies entirely and just jump right into building? is there enough cached information to do that?

[cunei]

@SethTisue You may be interested in the branch https://github.com/lightbend/dbuild/commits/wip-INPROGRESS-disableUpdates, which should be now complete. :blush: It is published as version 0.9.17-toni4, you can try it out now. It adds:

• an option '-s', aka '--skip-git-updates`, to bypass remote git updates
• huge improvements to git fetch operations, which by default now retrieve only the minimum amount of information for the specific branch/tag/PR requested
• an optional ?shallow=false uri query, to request a full clone when needed (see docs)

The speed improvements are substantial; when remote updates are disabled, the entire extraction phase for the community build (with unchanged references) takes about 25 seconds.

[cunei]

A matching update to the community build is at: https://github.com/cunei/community-builds/commits/wip-testing-toni4

[cunei]

@SethTisue I made further improvements and optimizations. Now, re-extracting the 188 projects of the community build, with all cached and updates disabled, takes just 8 seconds:

$ ./run.sh -lds 2>&1 | ts '[%T]'
[...]
[10:31:04] [acyclic] --== Extracting dependencies for acyclic ==--
[...]
[10:31:12] [zinc] --== End Extracting dependencies for zinc ==--
[10:31:12] [info] ---== Dependency Information ===---
[...]

You can test it by using dbuild version 0.9.17-toni22. The matching changes to the community buld (only tested up to the end of extraction) are at https://github.com/cunei/community-builds/commits/wip-testing-toni22

[ashawley]

an option '-s', aka '--skip-git-updates`, to bypass remote git updates

Why not wrap these various behaviors of Git and sbt skipping update in to something called --offline mode, as it is called in Maven and sbt? That way, you can run dbuild on plane rides.

Also, why not use --depth=1 on the initial git clone (maybe by default)? This would be a lot easier for first time users, reduce the disk storage requirements, and also avoid abusing GitHub.

[SethTisue]

I'll have to look at this in January

[SethTisue]

the narrow.sc script has pretty much removed this pain; you can use it so only the repos you care about at the moment are fetched or extracted