bazelbuild / rules_scala

Scala rules for Bazel
Apache License 2.0
362 stars 274 forks source link

Scala3 compiler crashes when cats-effect library is added as a dependency #1408

Open meisam opened 2 years ago

meisam commented 2 years ago

Steps to reproduce the crash:

  1. Add dependencies to cats-effect:
    maven_install(
    artifacts = [
        "org.typelevel:cats-free_3:2.8.0",
        "org.typelevel:cats-core_3:2.8.0",
    ],
    repositories = [
        "https://repo1.maven.org/maven2",
    ],
    )
  2. import the cats library in a scala file.
  3. Set the SCALA_VERIOS to 3 in scala_rules (eg. 3.1.0, or 3.1.1, or 3.1.2).
  4. Build the project with bazel's scala_rules.

The Scala3 compiler, dotty, crashes with:

exception occurred while compiling path/to/ReproDotytCrash.scala
scala.MatchError: val <none> (of class dotty.tools.dotc.core.Symbols$NoSymbol$) while compiling 
path/to/ReproDotytCrash.scala
val <none> (of class dotty.tools.dotc.core.Symbols$NoSymbol$)
scala.MatchError: val <none> (of class dotty.tools.dotc.core.Symbols$NoSymbol$)
        at dotty.tools.dotc.core.Types$ThisType.cls(Types.scala:2770)
        at dotty.tools.dotc.core.Types$ThisType.underlying(Types.scala:2775)
        at dotty.tools.dotc.core.Types$Type.goThis$1(Types.scala:806)
        at dotty.tools.dotc.core.Types$Type.go$1(Types.scala:700)
        at dotty.tools.dotc.core.Types$Type.findMember(Types.scala:870)
        at dotty.tools.dotc.core.Types$Type.memberBasedOnFlags(Types.scala:666)
        at dotty.tools.dotc.core.Types$Type.member(Types.scala:650)
        at dotty.tools.dotc.core.Types$NamedType.memberDenot(Types.scala:2269)
        at dotty.tools.dotc.core.Types$NamedType.memberDenot(Types.scala:2256)
        at dotty.tools.dotc.core.Types$NamedType.fromDesignator$1(Types.scala:2212)
        at dotty.tools.dotc.core.Types$NamedType.computeDenot(Types.scala:2236)
        at dotty.tools.dotc.core.Types$NamedType.denot(Types.scala:2191)
        at dotty.tools.dotc.core.Types$NamedType.computeSymbol(Types.scala:2148)
        at dotty.tools.dotc.core.Types$NamedType.symbol(Types.scala:2141)
        at dotty.tools.dotc.core.Types$Type.typeSymbol(Types.scala:484)
        at dotty.tools.dotc.core.tasty.TreeUnpickler.readAnnot$$anonfun$2$$anonfun$1(TreeUnpickler.scala:701)
        at dotty.tools.dotc.core.Annotations$.dotty$tools$dotc$core$Annotations$$anon$3$$_$$lessinit$greater$$anonfun$3(Annotations.scala:165)
        at dotty.tools.dotc.core.Annotations$LazyAnnotation.symbol(Annotations.scala:64)
        at dotty.tools.dotc.core.Annotations$Annotation.matches(Annotations.scala:22)
        at dotty.tools.dotc.core.SymDenotations$SymDenotation.dropOtherAnnotations(SymDenotations.scala:283)
        at dotty.tools.dotc.core.SymDenotations$SymDenotation.getAnnotation(SymDenotations.scala:245)
        at dotty.tools.dotc.core.SymDenotations$SymDenotation.targetName(SymDenotations.scala:526)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleExternalRef$1(TreePickler.scala:199)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleNewType(TreePickler.scala:220)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleType(TreePickler.scala:160)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleExternalRef$1(TreePickler.scala:200)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleNewType(TreePickler.scala:220)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleType(TreePickler.scala:160)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:375)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree$$anonfun$38$$anonfun$1(TreePickler.scala:428)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree$$anonfun$3(TreePickler.scala:428)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:429)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTreeUnlessEmpty(TreePickler.scala:319)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleDef$$anonfun$1(TreePickler.scala:335)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleDef(TreePickler.scala:337)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:545)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree$$anonfun$40$$anonfun$1(TreePickler.scala:469)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree$$anonfun$9(TreePickler.scala:469)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:469)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTreeUnlessEmpty(TreePickler.scala:319)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleDef$$anonfun$1(TreePickler.scala:335)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleDef(TreePickler.scala:337)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:560)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleStats$$anonfun$2(TreePickler.scala:360)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleStats(TreePickler.scala:360)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree$$anonfun$26(TreePickler.scala:586)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:587)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleDef$$anonfun$1(TreePickler.scala:332)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleDef(TreePickler.scala:337)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:562)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleStats$$anonfun$2(TreePickler.scala:360)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleStats(TreePickler.scala:360)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree$$anonfun$29(TreePickler.scala:602)
        at dotty.tools.dotc.core.tasty.TreePickler.withLength(TreePickler.scala:58)
        at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:602)
        at dotty.tools.dotc.core.tasty.TreePickler.pickle$$anonfun$1(TreePickler.scala:773)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.core.tasty.TreePickler.pickle(TreePickler.scala:773)
        at dotty.tools.dotc.transform.Pickler.run$$anonfun$3$$anonfun$2(Pickler.scala:69)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.transform.Pickler.run$$anonfun$1(Pickler.scala:106)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.transform.Pickler.run(Pickler.scala:106)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:308)
        at scala.collection.immutable.List.map(List.scala:246)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:309)
        at dotty.tools.dotc.transform.Pickler.runOn(Pickler.scala:111)
        at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:261)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
        at dotty.tools.dotc.Run.runPhases$5(Run.scala:272)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:280)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:289)
        at dotty.tools.dotc.Run.compileSources(Run.scala:222)
        at dotty.tools.dotc.Run.compile(Run.scala:206)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:39)
        at io.bazel.rulesscala.scalac.ScalacWorker3.compileScalaSources(ScalacWorker3.java:276)
        at io.bazel.rulesscala.scalac.ScalacWorker3.work(ScalacWorker3.java:83)
        at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:86)
        at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:39)
        at io.bazel.rulesscala.scalac.ScalacWorker3.main(ScalacWorker3.java:40)
Target //:repro-dotty-crash failed to build
INFO: Elapsed time: 0.578s, Critical Path: 0.43s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
meisam commented 2 years ago

I investigated this issue a bit and I have a hunch why it might be happening. It looks like ScalaWorker3.java, which is the class that packs the arguments for the Scala3 compiler and calls the compiler, is called with Scala2 libraries. To be more specific, if I patch ScalaWorker3.java and print the ScalaVersion during the build, i get scala version: 2.13.5.

diff --git a/src/java/io/bazel/rulesscala/scalac/ScalacWorker3.java b/src/java/io/bazel/rulesscala/scalac/ScalacWorker3.java
index 370be29..ef136ee 100644
--- a/src/java/io/bazel/rulesscala/scalac/ScalacWorker3.java
+++ b/src/java/io/bazel/rulesscala/scalac/ScalacWorker3.java
@@ -4,0 +5 @@ import scala.Tuple2;
+import scala.util.Properties;
@@ -75,0 +79 @@ class ScalacWorker3 implements Worker.Interface {
+        System.out.println("scala version: " + Properties.versionNumberString());

The compiler runs in the same process as ScalaWorker3.java, it is just a call to the doCompile method:

  Compiler compiler = driver.newCompiler(r._2);

  Reporter reporter = driver.doCompile(compiler, r._1, r._2);

It makes sense that a new process is not spawn for each new compile task, so pay the startup and warmup cost of the scala compiler once, but it sounds like the reported scala version at runtime should match the expected scala3 version.

Cats-effect is an involved library and exercises advanced features in the scala compiler. I thinks that's why this crash does not happen with all the libraries.

I'll do more tests and report the findings.

fabianhjr commented 1 year ago

Got a reproducible nix + bazel build

https://github.com/LibreCybernetics/Monorepo/tree/ref-bazelbuild-scala-3-typelevel-issue

  1. Checkout the above branch ( ref-bazelbuild-scala-3-typelevel-issue )
  2. nix develop --ignore-environment
  3. bazel build //code/lib/toml/core:toml-core

vs sbt

  1. cd code/
  2. sbt "toml-coreJVM/compile"

Relating to the comment on https://github.com/bazelbuild/rules_scala/pull/1465#issuecomment-1406636236

WojciechMazur commented 2 weeks ago

The reproducer seems to be failing due to missing dependency on cats-kernel. https://github.com/LibreCybernetics/Monorepo/blob/11a67bd75485439f256070be0337d8a77447e7b3/code/lib/toml/core/BUILD#L7

I'm new to Bazel, but as I understand this example for maven_install the project can add a top-level artifact to install the dependency, but each of transitive dependencies needs to be listed in the deps for given target.

I tested it locally using release 6.6.0 and main with Scala 3.2.1/3.3.1 and 3.3.3 respectively and was able to reproduce the compiler crash. Adding the missing transitive dependency fixed the problem.

I'll try to create a self-contained reproducer for the missing transitive dependency and we'll check if we can make the Scala compiler create a meaningful error instead of crashing the build.

I believe It allows to close this and the linked issues.