search

scala / scala3

The Scala 3 compiler, also known as Dotty.
https://dotty.epfl.ch
Apache License 2.0
5.81k stars 1.05k forks source link

Scaladoc generation fails when extending some Java-defined or Scala-2-defined classes #15927

Closed GitZinger closed 1 year ago

GitZinger commented 2 years ago

Compiler version

3.2.0.rc4 and sbt 1.7.1,

If you're not sure what version you're using, run print scalaVersion from sbt (if you're running scalac manually, use scalac -version instead).

Minimized code

sbt clean compile doc package
//the following is **BugTest.scala**

//::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
/** @author  Gitzinger
 *
 *  @title   `BugTest` for Handle Dates and Times
 */

package test4education

//::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
/** The `BugTest`  class 
 */
abstract class BugTest () extends Numeric [BugTest] with Serializable:

    //::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
    /**  a `BugTest` 
     */
    def what() = println("a bug")

end BugTest

Output (click arrow to expand)

```scala // TODO add output here [info] Main Scala API documentation to ../.../scala-3.2.0-RC4/api... Problem parsing src/main/scala/BugTest.scala:[159..167..529], documentation may not be generated. java.lang.AssertionError: assertion failed | => rat scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:11) at dotty.tools.dotc.util.Spans$Span$.start$extension(Spans.scala:45) at dotty.tools.dotc.util.SourcePosition.start(SourcePosition.scala:52) at scala.quoted.runtime.impl.QuotesImpl$reflect$PositionMethods$.start(QuotesImpl.scala:2813) at scala.quoted.runtime.impl.QuotesImpl$reflect$PositionMethods$.start(QuotesImpl.scala:2813) at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsTreeSymbolTuples$$anonfun$1(ClassLikeSupport.scala:260) at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:472) at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:576) at scala.collection.immutable.List.prependedAll(List.scala:152) at scala.collection.immutable.List$.from(List.scala:684) at scala.collection.immutable.List$.from(List.scala:681) at scala.collection.IterableOps$WithFilter.map(Iterable.scala:891) at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsTreeSymbolTuples(ClassLikeSupport.scala:264) at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsTreeSymbolTuples$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.getParentsAsTreeSymbolTuples(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsLinkToTypes(ClassLikeSupport.scala:253) at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsLinkToTypes$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.getParentsAsLinkToTypes(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass(ClassLikeSupport.scala:106) at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.mkClass(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike(ClassLikeSupport.scala:290) at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.parseClasslike(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseInheritedMember$$anonfun$1(ClassLikeSupport.scala:190) at dotty.tools.scaladoc.tasty.TastyParser.processTreeOpt(TastyParser.scala:204) at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseInheritedMember(ClassLikeSupport.scala:202) at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractMembers$$anonfun$2(ClassLikeSupport.scala:221) at scala.collection.immutable.List.flatMap(List.scala:293) at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractMembers(ClassLikeSupport.scala:221) at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractMembers$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.extractMembers(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractPatchedMembers(ClassLikeSupport.scala:226) at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractPatchedMembers$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.extractPatchedMembers(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass(ClassLikeSupport.scala:113) at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.mkClass(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike(ClassLikeSupport.scala:290) at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike$(ClassLikeSupport.scala:15) at dotty.tools.scaladoc.tasty.TastyParser.parseClasslike(TastyParser.scala:169) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.traverseTree(TastyParser.scala:223) at scala.quoted.Quotes$reflectModule$TreeTraverser.foldTree(Quotes.scala:4666) at scala.quoted.Quotes$reflectModule$TreeTraverser.foldTree$(Quotes.scala:4662) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldTree(TastyParser.scala:211) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldTree(TastyParser.scala:211) at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldTrees$$anonfun$1(Quotes.scala:4554) at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:169) at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:165) at scala.collection.immutable.List.foldLeft(List.scala:79) at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldTrees(Quotes.scala:4554) at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldTrees$(Quotes.scala:4549) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldTrees(TastyParser.scala:211) at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldOverTree(Quotes.scala:4617) at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldOverTree$(Quotes.scala:4549) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldOverTree(TastyParser.scala:211) at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTreeChildren(Quotes.scala:4668) at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTreeChildren$(Quotes.scala:4662) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.traverseTreeChildren(TastyParser.scala:211) at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTree(Quotes.scala:4664) at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTree$(Quotes.scala:4662) at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.traverseTree(TastyParser.scala:219) at dotty.tools.scaladoc.tasty.TastyParser.parseRootTree(TastyParser.scala:228) at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.$anonfun$4(TastyParser.scala:123) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.postProcess$$anonfun$2(TastyParser.scala:42) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) at scala.collection.immutable.List.foreach(List.scala:333) at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.postProcess(TastyParser.scala:42) at scala.tasty.inspector.OldTastyInspector$TastyInspectorFinishPhase$1.runOn(OldTastyInspector.scala:91) at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:234) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1328) at dotty.tools.dotc.Run.runPhases$1(Run.scala:245) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:253) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:262) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:262) at dotty.tools.dotc.Run.compileUnits(Run.scala:192) at dotty.tools.dotc.fromtasty.TASTYRun.compile(TASTYRun.scala:14) at dotty.tools.dotc.Driver.doCompile(Driver.scala:35) at dotty.tools.dotc.Driver.process(Driver.scala:195) at scala.tasty.inspector.OldTastyInspector.inspectFilesInContext(OldTastyInspector.scala:72) at scala.tasty.inspector.OldTastyInspector.inspectFilesInContext$(OldTastyInspector.scala:22) at scala.tasty.inspector.DocTastyInspector.inspectFilesInContext(DocTastyInspector.scala:5) at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.result(TastyParser.scala:147) at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule(ScalaModuleProvider.scala:11) at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:230) at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:72) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) at scala.Option.map(Option.scala:242) at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:76) at dotty.tools.dottydoc.Main$.process(Main.scala:25) at dotty.tools.dottydoc.Main.process(Main.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at xsbt.DottydocRunner.run(DottydocRunner.java:61) at xsbt.ScaladocInterface.run(ScaladocInterface.java:11) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329) at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:175) at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:133) at sbt.Doc$.$anonfun$scaladoc$1(Doc.scala:52) at sbt.Doc$.$anonfun$scaladoc$1$adapted(Doc.scala:40) at sbt.RawCompileLike$.$anonfun$prepare$1(RawCompileLike.scala:79) at sbt.RawCompileLike$.$anonfun$prepare$1$adapted(RawCompileLike.scala:72) at sbt.RawCompileLike$.$anonfun$cached$4(RawCompileLike.scala:63) at sbt.RawCompileLike$.$anonfun$cached$4$adapted(RawCompileLike.scala:61) at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219) at sbt.RawCompileLike$.$anonfun$cached$1(RawCompileLike.scala:68) at sbt.RawCompileLike$.$anonfun$cached$1$adapted(RawCompileLike.scala:52) at sbt.Defaults$.$anonfun$docTaskSettings$4(Defaults.scala:2157) at scala.Function1.$anonfun$compose$1(Function1.scala:49) at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62) at sbt.std.Transform$$anon$4.work(Transform.scala:68) at sbt.Execute.$anonfun$submit$2(Execute.scala:282) at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23) at sbt.Execute.work(Execute.scala:291) at sbt.Execute.$anonfun$submit$1(Execute.scala:282) at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) at sbt.CompletionService$$anon$2.call(CompletionService.scala:64) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) [warn] Companion for class xxx exists but is missing in classlike map ```
SethTisue commented 2 years ago

How would we reproduce the problem on our own computers?

What is the minimum code xxx.scala must contain for the bug to be triggered?

GitZinger commented 2 years ago

How would we reproduce the problem on our own computers?

What is the minimum code xxx.scala must contain for the bug to be triggered?

@SethTisue Hey sorry about that. I just updated the codes. if you run sbt clean compile doc package on a project with codes like this, you will end up with the same issue.

Could you look into it please? I can even push a repo with this the simplified code.

SethTisue commented 2 years ago

Could you look into it please?

It's not my area of expertise, but perhaps one of the Scaladoc maintainers will have a look.

SethTisue commented 2 years ago

You've improved the report greatly by supplying code, but presumably most of that code presumably isn't actually necessary for reproducing the bug. The report would be even higher quality, and much likelier to be acted upon, if you reduced it to the absolute minimum necessary code to demonstrate the problem. (And that's why Paweł added the "needs minimization" label.)

GitZinger commented 2 years ago

You've improved the report greatly by supplying code, but presumably most of that code presumably isn't actually necessary for reproducing the bug. The report would be even higher quality, and much likelier to be acted upon, if you reduced it to the absolute minimum necessary code to demonstrate the problem. (And that's why Paweł added the "needs minimization" label.)

@SethTisue Thanks for the advice. Just updated it. Please help me out.

prolativ commented 2 years ago

A slightly simpler minimization:

trait Foo extends Numeric[Any]

It crashes with just sbt clean doc - no package required.

GitZinger commented 2 years ago

A slightly simpler minimization:

trait Foo extends Numeric[Any]

It crashes with just sbt clean doc - no package required.

@SethTisue @prolativ Thanks. Are you looking into it? or is it already fixed with the next release version?

prolativ commented 2 years ago

Currently I have other priorities so I cannot give you any guarantee about when this would be fixed. For now this looks like a very strange corner case. I played with this example a bit and didn't manage to make this crash when Numeric gets replaced with some other type

SethTisue commented 2 years ago

is it already fixed with the next release version?

you can find that out for yourself by trying it in the latest nightly build

GitZinger commented 2 years ago

Currently I have other priorities so I cannot give you any guarantee about when this would be fixed. For now this looks like a very strange corner case. I played with this example a bit and didn't manage to make this crash when Numeric gets replaced with some other type

There are a lot of other types that can cause this too. For instance,

import javax.swing.JPanel

class bug2 extends JPanel:

end bug2

and with sbt clean doc this will cause 

scala-3.2.0-RC4/api...
Problem parsing src/main/scala/Bug2.scala:<0..65>, documentation may not be generated.
java.lang.AssertionError: assertion failed
  | => rat scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:11)
    at dotty.tools.dotc.util.Spans$Span$.start$extension(Spans.scala:45)
    at dotty.tools.dotc.util.SourcePosition.start(SourcePosition.scala:52)
    at scala.quoted.runtime.impl.QuotesImpl$reflect$PositionMethods$.start(QuotesImpl.scala:2813)
    at scala.quoted.runtime.impl.QuotesImpl$reflect$PositionMethods$.start(QuotesImpl.scala:2813)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsTreeSymbolTuples$$anonfun$1(ClassLikeSupport.scala:260)
    at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:472)
    at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:576)
    at scala.collection.immutable.List.prependedAll(List.scala:152)
    at scala.collection.immutable.List$.from(List.scala:684)
    at scala.collection.immutable.List$.from(List.scala:681)
    at scala.collection.IterableOps$WithFilter.map(Iterable.scala:891)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsTreeSymbolTuples(ClassLikeSupport.scala:264)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsTreeSymbolTuples$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.getParentsAsTreeSymbolTuples(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsLinkToTypes(ClassLikeSupport.scala:253)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.getParentsAsLinkToTypes$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.getParentsAsLinkToTypes(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass(ClassLikeSupport.scala:106)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.mkClass(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike(ClassLikeSupport.scala:290)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.parseClasslike(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseInheritedMember$$anonfun$1(ClassLikeSupport.scala:190)
    at dotty.tools.scaladoc.tasty.TastyParser.processTreeOpt(TastyParser.scala:204)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseInheritedMember(ClassLikeSupport.scala:202)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractMembers$$anonfun$2(ClassLikeSupport.scala:221)
    at scala.collection.immutable.List.flatMap(List.scala:293)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractMembers(ClassLikeSupport.scala:221)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractMembers$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.extractMembers(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractPatchedMembers(ClassLikeSupport.scala:226)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.extractPatchedMembers$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.extractPatchedMembers(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass(ClassLikeSupport.scala:113)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.mkClass$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.mkClass(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike(ClassLikeSupport.scala:290)
    at dotty.tools.scaladoc.tasty.ClassLikeSupport.parseClasslike$(ClassLikeSupport.scala:15)
    at dotty.tools.scaladoc.tasty.TastyParser.parseClasslike(TastyParser.scala:169)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.traverseTree(TastyParser.scala:223)
    at scala.quoted.Quotes$reflectModule$TreeTraverser.foldTree(Quotes.scala:4666)
    at scala.quoted.Quotes$reflectModule$TreeTraverser.foldTree$(Quotes.scala:4662)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldTree(TastyParser.scala:211)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldTree(TastyParser.scala:211)
    at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldTrees$$anonfun$1(Quotes.scala:4554)
    at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:169)
    at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:165)
    at scala.collection.immutable.List.foldLeft(List.scala:79)
    at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldTrees(Quotes.scala:4554)
    at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldTrees$(Quotes.scala:4549)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldTrees(TastyParser.scala:211)
    at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldOverTree(Quotes.scala:4617)
    at scala.quoted.Quotes$reflectModule$TreeAccumulator.foldOverTree$(Quotes.scala:4549)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.foldOverTree(TastyParser.scala:211)
    at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTreeChildren(Quotes.scala:4668)
    at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTreeChildren$(Quotes.scala:4662)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.traverseTreeChildren(TastyParser.scala:211)
    at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTree(Quotes.scala:4664)
    at scala.quoted.Quotes$reflectModule$TreeTraverser.traverseTree$(Quotes.scala:4662)
    at dotty.tools.scaladoc.tasty.TastyParser$Traverser$2$.traverseTree(TastyParser.scala:219)
    at dotty.tools.scaladoc.tasty.TastyParser.parseRootTree(TastyParser.scala:228)
    at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.$anonfun$4(TastyParser.scala:123)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.postProcess$$anonfun$2(TastyParser.scala:42)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.postProcess(TastyParser.scala:42)
    at scala.tasty.inspector.OldTastyInspector$TastyInspectorFinishPhase$1.runOn(OldTastyInspector.scala:91)
    at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:234)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1328)
    at dotty.tools.dotc.Run.runPhases$1(Run.scala:245)
    at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:253)
    at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:262)
    at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68)
    at dotty.tools.dotc.Run.compileUnits(Run.scala:262)
    at dotty.tools.dotc.Run.compileUnits(Run.scala:192)
    at dotty.tools.dotc.fromtasty.TASTYRun.compile(TASTYRun.scala:14)
    at dotty.tools.dotc.Driver.doCompile(Driver.scala:35)
    at dotty.tools.dotc.Driver.process(Driver.scala:195)
    at scala.tasty.inspector.OldTastyInspector.inspectFilesInContext(OldTastyInspector.scala:72)
    at scala.tasty.inspector.OldTastyInspector.inspectFilesInContext$(OldTastyInspector.scala:22)
    at scala.tasty.inspector.DocTastyInspector.inspectFilesInContext(DocTastyInspector.scala:5)
    at dotty.tools.scaladoc.tasty.ScaladocTastyInspector.result(TastyParser.scala:147)
    at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule(ScalaModuleProvider.scala:11)
    at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:230)
    at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:72)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.Option.map(Option.scala:242)
    at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:76)
    at dotty.tools.dottydoc.Main$.process(Main.scala:25)
    at dotty.tools.dottydoc.Main.process(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at xsbt.DottydocRunner.run(DottydocRunner.java:61)
    at xsbt.ScaladocInterface.run(ScaladocInterface.java:11)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329)
    at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:175)
    at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:133)
    at sbt.Doc$.$anonfun$scaladoc$1(Doc.scala:52)
    at sbt.Doc$.$anonfun$scaladoc$1$adapted(Doc.scala:40)
    at sbt.RawCompileLike$.$anonfun$prepare$1(RawCompileLike.scala:79)
    at sbt.RawCompileLike$.$anonfun$prepare$1$adapted(RawCompileLike.scala:72)
    at sbt.RawCompileLike$.$anonfun$cached$4(RawCompileLike.scala:63)
    at sbt.RawCompileLike$.$anonfun$cached$4$adapted(RawCompileLike.scala:61)
    at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
    at sbt.RawCompileLike$.$anonfun$cached$1(RawCompileLike.scala:68)
    at sbt.RawCompileLike$.$anonfun$cached$1$adapted(RawCompileLike.scala:52)
    at sbt.Defaults$.$anonfun$docTaskSettings$4(Defaults.scala:2157)
    at scala.Function1.$anonfun$compose$1(Function1.scala:49)
    at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
    at sbt.std.Transform$$anon$4.work(Transform.scala:68)
    at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
    at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
    at sbt.Execute.work(Execute.scala:291)
    at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
    at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
[info] Main Scala API documentation successful.

Yeah there are more.. @prolativ I really appreciate it. Take time for your high priority stuff. Thanks a lot

SethTisue commented 2 years ago

One thing I wonder is if the parent must be Java-defined to trigger the bug.

Not every Java-defined parent triggers it, though (for example, the problem goes away if you change JPanel to JFrame).

@anatoliykmetyuk might be a good spree ticket?

SethTisue commented 1 year ago

reported again at https://github.com/lampepfl/dotty/issues/16290 (on extends Iterator[String])

godenji commented 1 year ago

I'm seeing this as well, with traits/abstract classes extending java.io.Closeable.

Same stacktrace as above:

Scala 3.2.2-RC1 sbt 1.8.0 OpenJDK 19

SethTisue commented 1 year ago

reported again at #16546 and again at #16547

maybe duplicate, maybe not: #16180

scala-center-bot commented 1 year ago

This issue was picked for the Issue Spree No. 25 of 24 January 2023 which takes place in a week from now. @SethTisue, @jan-pieter, @yzia2000 will be working on it. If you have any insight into the issue or guidance on how to fix it, please leave it here.

SethTisue commented 1 year ago

When preparing for a spree, one tries to strike a balance between being unprepared, and over-preparing in the sense that you figure out what's going on before the spree has even started :-)

I wasn't sure how the Scaladoc tool was tested, and there didn't seem to be a relevant directory under tests.

so I searched recently merged PRs for "Scaladoc": https://github.com/lampepfl/dotty/pulls?q=is%3Apr+sort%3Aupdated-desc+is%3Amerged+scaladoc since presumably these PRs would touch test cases

and indeed, that led me to discover the scaladoc-testcases directory, which is at the top level rather than under tests

then I wondered how they were run, and git grep scaladoc-testcases led me to build.sbt where there's a scaladoc-testcases subproject. I tried scaladoc-testcases/test but that didn't do anything. hmm, what to try next?

after further poking around I found scaladoc/README.md, but it still wasn't absolutely clear to me, after reading that, how the scaladoc-testcases functions. I thought I'd look some more at a recent PR that did a bug fix to the Scaladoc tool, so I looked at https://github.com/lampepfl/dotty/pull/14810 and found an exchange between Julien and Quentin (@Sporarum) where Quentin says:

This works by comparing the text of the file with the output of scala-meta, similarly to what is done in the compiler with the tests/pos, tests/neg folders. This is therefore both the declaration and the specification for the tests !

wow, cool! but still not clear to me how to run the tests?

as a guess, maybe scaladoc/test? that seems promising; the output contains some references to the scaladoc-testcases directory

so I tried creating scaladoc-testcases/src/tests/numeric.scala containing trait Foo extends Numeric[Any] , hoping scaladoc/test would fail... but it didn't

another thing that git grep scaladoc-testcases turned up was the existence of project/scripts/cmdScaladocTests, so I tried running that script but it didn't fail 🤔

perhaps I'm expected to somehow tell the testing stuff that I added a file, maybe files in scaladoc-testcases/src/tests aren't picked up automatically? I tried git greping for the names of some of the source files there and I did find that they seem to explicitly be mentioned in source files under scaladoc/test/dotty/tools/scaladoc

but it's not clear to me in this case where the right place to add my new file would be. maybe to scaladoc/test/dotty/tools/scaladoc/signatures/TranslatableSignaturesTestCases.scala, again based on looking at what was done in #14810

so at the end of that file I added:

class Numeric extends SignatureTest("numeric", SignatureTest.all)

and I ran scaladoc/testOnly *scaladoc.signatures.Numeric

and it said "The test name is incorrect or scaladoc-testcases were not recompiled"

I looked at another test file in the same directory and I noticed it started with a package declaration, so I tried changing numeric.scala to include one too:

package tests
package numeric

trait Foo extends Numeric[Any]

and lo and behold, now when I run scaladoc/testOnly *scaladoc.signatures.Numeric, it fails with "Problem parsing scaladoc-testcases/src/tests/numeric.scala:[0..8..61], documentation may not be generated", which is the desired mode of failure

and I also verified that if I changed the test file to something harmless like extends AnyRef, that it doesn't fail

so, I think at this point we know one way to add a test for this — maybe not the best way, but enough to proceed with attempting a fix at the spree

@jan-pieter it looks like you have one previous contribution here, and @yzia2000 it looks like it's your first time? though I might be mistaken? if I'm right, we should let @yzia2000 drive to make sure he gets up to speed on basics of contributing

SethTisue commented 1 year ago

One thing that's annoying here is that in the original bug report we get a stack trace that shows where it's blowing up, but in the context of scaladoc/testOnly *scaladoc.signatures.Numeric, we don't get that stack trace, we only get a stack trace from within dotty.tools.scaladoc.signatures.SignatureTest itself.

SethTisue commented 1 year ago

One thing we might want to try to narrow down here is, are the contents of Foo.tasty correct and we need to fix the reader? Or does Foo.tasty actually contain incorrect span information, and we need to fix it at the point where the incorrect information is written?

I feel like I've seen a way to dump out the contents of a TASTy file in human-readable form, but I don't remember offhand what it is.

Anyway, that's as much time as I have for preparation. I'm starting to get a little worried this ticket is a bit difficult for the spree, but that's okay, we can try to make some headway on it regardless.

SethTisue commented 1 year ago

@Sporarum @szymon-rd @pikinier20 any pointers for us...?

jan-pieter commented 1 year ago

This is the exact same process I went through yesterday and with the same result :joy:. It indeed seems quite a complex issue but let's see how far we get. Looking at the stack trace poster earlier and the code I was wondering whether it could be an issue in the parents of a class. Regarding printing tasty files: that seems to be part of the compiler

SethTisue commented 1 year ago

We believe the problem is in ClassLikeSupport.scala, in this line:

        parentTree <- c.parents if parentTree.pos.start != parentTree.pos.end // We assume here that order is correct

if .pos doesn't exist (is NoSpan), call .start and .end blow up

What we want is to call .exists first, but .exists doesn't exist in the PositionMethods trait in Quotes.scala. It should exist, but we can't add it until 3.4, since 3.3.0-RC1 is already imminent.

In the long run, we should add probably add .exists to the API, but it would be nice to have a shorter-term fix in the meantime since 3.4 is a ways off and this bug is being frequently reported.

SethTisue commented 1 year ago

Caveat: not crashing isn't enough, we also need to verify that the parent still shows up in the generated Scaladoc.

SethTisue commented 1 year ago

One thing I wonder is if the parent must be Java-defined to trigger the bug

No, because for example Numericand Iterator aren't Java-defined, they're Scala-2-defined (in the Scala 2.13 standard library). But I don't think we have a failure example where the parent is Scala-3-defined.

SethTisue commented 1 year ago

in the sbt build, scaladoc/run is how we can manually run Scaladoc against a TASTy file — I hadn't found that before

SethTisue commented 1 year ago

16759 is the short-term fix which we can backport to the 3.3.x series

16761 is the better long-term fix

julienrf commented 1 year ago

Minor comment about wording: would isDefined: Boolean be more consistent with the standard library? (like isDefined on Option. exists usually takes a predicate as a parameter).

SethTisue commented 1 year ago

Minor comment about wording: would isDefined: Boolean be more consistent with the standard library

good comment/question, but: .exists is widely used within the compiler (namely, on symbols and types), and it's probably more important to be consistent with that, especially since this isn't a brand new method; we're only making an existing method visible via an interface

Sporarum commented 1 year ago

@SethTisue

wow, cool! but still not clear to me how to run the tests?

I lived through the same, so this seems to indicate some documentation is missing

I had actually isolated a commit to make it only about setting up the test: https://github.com/lampepfl/dotty/pull/14321/commits/5671d26590b4cd9bc7a8f0858022e6947f622fea

Something that pointed in the right direction was the old behaviour of running test in dotty, it used to say something like:

Running all the tests takes a very long time, it's probably not what you want to do, instead you can use:
testCompilation - runs all compile tests
someWayToStartTheScaladocTests - runs all scaladoc tests
...

I don't know why the behavior was changed

I do not have time at the moment, do you think you could investigate what makes it hard to start the scaladoc tests, and address it ?

(Probably through creating some documentation in the readme or in https://docs.scala-lang.org/scala3/guides/contribution/procedures-intro.html, in particular https://docs.scala-lang.org/scala3/guides/contribution/procedures-testing.html does not mention the commands to test other things like scaladoc)

nicolasstucki commented 1 year ago

The tree that has a non-existent position should have a position.

nicolasstucki commented 1 year ago

The way to debug this is to set -Ydebug-tree-with-id -2 and figure out the uniqueId of the tree that does not have a position. If the id is 234, re-run with -Ydebug-tree-with-id 234. This will print the stack trace of the location where the tree was created. If it is part of a tree copy, we need to figure out the id of the original tree without a position and repeat. There is probably a tree for which we are forgetting to set the span (withSpan).

nicolasstucki commented 1 year ago

It seems that the problematic trees are generated here: https://github.com/lampepfl/dotty/blob/main/compiler/src/dotty/tools/dotc/quoted/reflect/FromSymbol.scala#L33

We should probably set the span of those trees to cls.span.

SethTisue commented 1 year ago

I lived through the same, so this seems to indicate some documentation is missing

(working on it over at https://github.com/lampepfl/dotty/pull/16769)

SethTisue commented 1 year ago

Thanks @nicolasstucki for digging into the underlying cause. I'll get back to it (but not right away).