search

scalacenter / scala-debug-adapter

Implementation of the Debug Adapter Protocol for Scala
Apache License 2.0
57 stars 26 forks source link

Cache the `ClassEntryLookUp`s #81

Open adpi2 opened 3 years ago

adpi2 commented 3 years ago

Problem

Building the SourceLookUpProvider takes time because it must go through all the source files and all the class files of the project, its dependencies and the java runtime. This operation is splitted in n where n is the number of class path entries plus one for the java runtime. Internally, the project class directory and its sources, a dependency jar with its source jar, the java runtime and its src.zip file are called ClassEntries.

Current stats (just before 2.0.0) ```scala + adopt:1.8.0-292: - 20069 classes loaded in 1107 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.8.0-292 4178ms adopt:1.9.0-0: - 36666 classes loaded in 1504 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.9.0-0 10733ms adopt:1.10.0-2: - 35853 classes loaded in 1260 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.10.0-2 6285ms adopt:1.11.0-11: - 30395 classes loaded in 839 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.11.0-11 5426ms adopt:1.12.0-2: - 30328 classes loaded in 821 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.12.0-2 6974ms adopt:1.13.0-2: - 30439 classes loaded in 895 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.13.0-2 6771ms adopt:1.14.0-2: - 30633 classes loaded in 810 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.14.0-2 4335ms adopt:1.15.0-2: - 29285 classes loaded in 844 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.adopt:1.15.0-2 5184ms scala-library: - 2870 classes loaded in 211 milliseconds scala-compiler: - 3399 classes loaded in 284 milliseconds - 3 orphan class files (0.08%) scala3-library_3: - 487 classes loaded in 35 milliseconds scala3-compiler_3: - 3565 classes loaded in 483 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.scala-lang 2006ms scalatest-core_3: - 1282 classes loaded in 81 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.scalatest 278ms cats-core_3: - 2330 classes loaded in 134 milliseconds cats-kernel_3: - 1020 classes loaded in 68 milliseconds shapeless_2.13: - 2228 classes loaded in 207 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.typelevel 878ms main_2.12: - 911 classes loaded in 68 milliseconds lm-coursier-shaded_2.12: - 3972 classes loaded in 213 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.sbt 2155ms spark-core_2.12: - 3764 classes loaded in 244 milliseconds - 1 orphan class files (0.02%) spark-catalyst_2.12: - 3654 classes loaded in 244 milliseconds spark-sql_2.12: - 2306 classes loaded in 179 milliseconds spark-mllib_2.12: - 2302 classes loaded in 151 milliseconds - 2 orphan class files (0.08%) + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.spark 7188ms akka-actor_2.13: - 1793 classes loaded in 91 milliseconds akka-stream_2.13: - 2535 classes loaded in 127 milliseconds akka-http-core_2.13: - 2084 classes loaded in 132 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.akka 873ms zio_3: - 810 classes loaded in 56 milliseconds zio-streams_3: - 183 classes loaded in 21 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.zio 229ms guava: - 2030 classes loaded in 91 milliseconds + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.guava 236ms play_2.13: - 1383 classes loaded in 128 milliseconds - 2 orphan class files (0.14%) + ch.epfl.scala.debugadapter.internal.ClassEntryLookUpStats.play 678ms scalaz-core_2.13: - 3507 classes loaded in 280 milliseconds ```

Each time a new debug session is started, it recomputes all the ClassEntryLookUps which takes a few seconds (maybe up to 10-20 seconds on big projects and slow machine).

Solution

We can cache the ClassEntryLookUps so that:

In particular the scala-library lookup and the java runtime lookup should never be recomputed.

Implementation

A ClassEntryLookUp could be indexed by the path of its jar file or class directory. A ClassEntryLookUp could be invalidated based on the hash of the jar file or class directory. As long as the debug server is started by the build tool (sbt or Bloop) this hash could be the Zinc hash. When the debug server will be started by Metals, if ever, we will need to find another solution. The cache could be a static Map in the scala-debug-adapter, or it could be managed by the build tool. What would be the advantage of a managed cache? IDK

tgodzik commented 3 years ago

Any chance we could calculate this at the same time we are indexing sources in Metals?

adpi2 commented 3 years ago

Yes if we ever manage to move the debug server to Metals.

The ClassEntryLookup must be recomputed if the project is re-compiled. So is it better to do it after each compilation (that could be too much) or just before starting the debug session start? IDK.

tgodzik commented 3 years ago

I think it's better to do it on recompilation, so that run/debug is fast from the point of the user. Similar to indexing semanticdb, which we also do on recompilation.