search

scala / scala3

The Scala 3 compiler, also known as Dotty.
https://dotty.epfl.ch
Apache License 2.0
5.81k stars 1.05k forks source link

scaladoc hangs #17577

Closed reid-spencer closed 1 year ago

reid-spencer commented 1 year ago

Compiler version

3.2.2 3.3.0-RC5 3.3.0-RC6 3.3.0 3.3.1-RC1

Minimized code

Not available yet. Extracting from 100s of classes is time-consuming.

Follow my analysis here: https://contributors.scala-lang.org/t/scala3doc-spin-loop/6202/7

Output

None other than:

[info] Main Scala API documentation to /Users/reid/Code/reactific/riddl/language/target/scala-3.3.0/api...

Expectation

The compiler completes the documentation generation instead of hanging forever.

reid-spencer commented 1 year ago

To reproduce:

That process will never end and a java process will run at > 100% CPU utilization forever (many hours tested).

reardonj commented 1 year ago

I took a bit of a look at this and have a minimization. I started by getting a thread dump while generating docs with from the affected code. Scalac appears to be getting stuck in the InheritanceInformationTransformer code of scaladoc generation, performing concatenation and distinct calls on Seq. I haven't delved into the scalac code yet, but it smells like an exponential blowup when there is a large amount of inheritance. The affected code includes a deep inheritance heirarchy with a series of traits, each defining inner traits, classes, and objects. Changing the design to use self types instead allows the documentation generation to complete.

This minimization only takes several minutes instead of hanging indefinitely, but hopefully gets the idea across compactly: https://github.com/reardonj/minimize-dotty-bug-17577

"pool-77-thread-11" #610 prio=5 os_prio=31 cpu=80053.92ms elapsed=83.12s tid=0x00007fa40db38800 nid=0xaf67 runnable  [0x0000700013958000]
   java.lang.Thread.State: RUNNABLE
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.runtime.ScalaRunTime$._hashCode(ScalaRunTime.scala:158)
        at dotty.tools.scaladoc.Kind$Trait.hashCode(api.scala:60)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.runtime.ScalaRunTime$._hashCode(ScalaRunTime.scala:158)
        at dotty.tools.scaladoc.LinkToType.hashCode(api.scala:141)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.Tuple2.hashCode(Tuple2.scala:24)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.collection.mutable.HashSet.add(HashSet.scala:69)
        at scala.collection.immutable.StrictOptimizedSeqOps.distinctBy(StrictOptimizedSeqOps.scala:35)
        at scala.collection.immutable.StrictOptimizedSeqOps.distinctBy$(StrictOptimizedSeqOps.scala:26)
        at scala.collection.immutable.List.distinctBy(List.scala:79)
        at scala.collection.SeqOps.distinct(Seq.scala:206)
        at scala.collection.SeqOps.distinct$(Seq.scala:206)
        at scala.collection.AbstractSeq.distinct(Seq.scala:1188)
        at dotty.tools.scaladoc.HierarchyGraph.$plus(api.scala:146)
        at dotty.tools.scaladoc.HierarchyGraph.$plus$plus$$anonfun$1(api.scala:148)
        at dotty.tools.scaladoc.HierarchyGraph$$Lambda$6734/0x0000000801cf2040.apply(Unknown Source)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.scaladoc.HierarchyGraph.$plus$plus(api.scala:149)
        at dotty.tools.scaladoc.api$package$.withNewGraphEdges(api.scala:220)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply$$anonfun$1(InheritanceInformationTransformer.scala:10)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer$$Lambda$6910/0x0000000801dca040.apply(Unknown Source)
        at dotty.tools.scaladoc.api$package$.updateRecusivly(api.scala:205)
        at dotty.tools.scaladoc.api$package$.$anonfun$1(api.scala:204)
        at dotty.tools.scaladoc.api$package$$$Lambda$6913/0x0000000801dcc840.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:250)
        at scala.collection.immutable.List.map(List.scala:79)
        at dotty.tools.scaladoc.api$package$.updateRecusivly(api.scala:204)
        at dotty.tools.scaladoc.api$package$.updateMembers$$anonfun$1$$anonfun$1(api.scala:240)
        at dotty.tools.scaladoc.api$package$$$Lambda$6912/0x0000000801dcb840.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:250)
        at scala.collection.immutable.List.map(List.scala:79)
        at dotty.tools.scaladoc.api$package$.updateMembers$$anonfun$1(api.scala:240)
        at dotty.tools.scaladoc.api$package$$$Lambda$6911/0x0000000801dcb040.apply(Unknown Source)
        at dotty.tools.scaladoc.api$package$.updatePackages(api.scala:236)
        at dotty.tools.scaladoc.api$package$.updateMembers(api.scala:240)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply(InheritanceInformationTransformer.scala:10)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply(InheritanceInformationTransformer.scala:5)
        at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule$$anonfun$1(ScalaModuleProvider.scala:51)
        at dotty.tools.scaladoc.ScalaModuleProvider$$$Lambda$6894/0x0000000801db7840.apply(Unknown Source)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule(ScalaModuleProvider.scala:51)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:241)
        at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:73)
        at dotty.tools.scaladoc.Scaladoc$$$Lambda$6622/0x0000000801c67040.applyVoid(Unknown Source)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.Option.map(Option.scala:242)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:77)
        at dotty.tools.dottydoc.Main$.process(Main.scala:25)
        at dotty.tools.dottydoc.Main.process(Main.scala)
        at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
        at xsbt.DottydocRunner.run(DottydocRunner.java:61)
        at xsbt.ScaladocInterface.run(ScaladocInterface.java:11)
        at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
        at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329)
        at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:175)
        at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:133)
        at sbt.Doc$.$anonfun$scaladoc$1(Doc.scala:52)
        at sbt.Doc$.$anonfun$scaladoc$1$adapted(Doc.scala:40)
        at sbt.Doc$$$Lambda$6575/0x0000000801c35c40.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$prepare$1(RawCompileLike.scala:79)
        at sbt.RawCompileLike$.$anonfun$prepare$1$adapted(RawCompileLike.scala:72)
        at sbt.RawCompileLike$$$Lambda$6576/0x0000000801c36040.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$cached$4(RawCompileLike.scala:63)
        at sbt.RawCompileLike$.$anonfun$cached$4$adapted(RawCompileLike.scala:61)
        at sbt.RawCompileLike$$$Lambda$6584/0x0000000801c3d840.apply(Unknown Source)
        at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
        at sbt.util.Tracked$$$Lambda$2940/0x0000000800db9840.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$cached$1(RawCompileLike.scala:68)
        at sbt.RawCompileLike$.$anonfun$cached$1$adapted(RawCompileLike.scala:52)
        at sbt.RawCompileLike$$$Lambda$6577/0x0000000801c36440.apply(Unknown Source)
        at sbt.Defaults$.$anonfun$docTaskSettings$4(Defaults.scala:2159)
        at sbt.Defaults$$$Lambda$811/0x0000000800640040.apply(Unknown Source)
        at scala.Function1.$anonfun$compose$1(Function1.scala:49)
        at scala.Function1$$Lambda$339/0x000000080040b040.apply(Unknown Source)
        at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
        at sbt.internal.util.$tilde$greater$$Lambda$2773/0x0000000800d53840.apply(Unknown Source)
        at sbt.std.Transform$$anon$4.work(Transform.scala:68)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
        at sbt.Execute$$Lambda$2799/0x0000000800d60c40.apply(Unknown Source)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
        at sbt.Execute.work(Execute.scala:291)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
        at sbt.Execute$$Lambda$2783/0x0000000800d59440.apply(Unknown Source)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
        at sbt.ConcurrentRestrictions$$anon$4$$Lambda$2796/0x0000000800d5f840.apply(Unknown Source)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
        at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
        at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.11/Executors.java:515)
        at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)