search

sbt / zinc

Scala incremental compiler library, used by sbt and other build tools
Apache License 2.0
334 stars 120 forks source link

Pipelining prevents Java compilation with compileOrder := JavaThenScala config #1352

Closed guersam closed 2 weeks ago

guersam commented 5 months ago

Original submitted as https://github.com/sbt/sbt/issues/7546, but I think it's a Zinc issue.

steps

  1. Checkout https://github.com/guersam/sbt-debug-pipelined-hybrid-project.
  2. Run $ sbt run.

problem

It fails due to a missing Java class. The incremental compilation log shows that the Java source code is not compiled.

expectation

It prints:

Hello world!
42

notes

  1. Both usePipelining := true and compileOrder := CompilerOrder.JavaThenScala settings are needed to reproduce
  2. It's reproducible both in sbt 1.9.9 and 1.10.0-RC2.
  3. It happens with both Scala 2 and 3.
  4. CI result with sbt 'last compileIncremental' log
[info] welcome to sbt 1.10.0-RC2 (Eclipse Adoptium Java 11.0.22)
[info] loading project definition from /home/runner/work/sbt-debug-pipelined-hybrid-project/sbt-debug-pipelined-hybrid-project/project
[info] loading settings for project root from build.sbt ...
[info] set current project to sbt-pipelining-debug (in build file:/home/runner/work/sbt-debug-pipelined-hybrid-project/sbt-debug-pipelined-hybrid-project/)
[debug] [zinc] IncrementalCompile -----------
[debug] IncrementalCompile.incrementalCompile
[debug] previous = Stamps for: 0 products, 0 sources, 0 libraries
[debug] current source = Set(${BASE}/src/main/java/example/JavaClass.java, ${BASE}/src/main/scala/example/Main.scala)
[debug] > initialChanges = InitialChanges(Changes(added = Set(${BASE}/src/main/java/example/JavaClass.java, ${BASE}/src/main/scala/example/Main.scala), removed = Set(), changed = Set(), unmodified = ...),Set(),Set(),API Changes: Set())
[debug] Full compilation, no sources in previous analysis.
[debug] all 2 sources are invalidated
[debug] Initial set of included nodes: 
[debug] Recompiling all sources: number of invalidated sources > 50.0 percent of all sources
[debug] compilation cycle 1
[info] compiling 1 Scala source and 1 Java source to /home/runner/work/sbt-debug-pipelined-hybrid-project/sbt-debug-pipelined-hybrid-project/target/scala-3.4.1/classes ...
[debug] Returning already retrieved and compiled bridge: /home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-sbt-bridge/3.4.1/scala3-sbt-bridge-3.4.1.jar.
[debug] [zinc] Running cached compiler 5[9](https://github.com/guersam/sbt-debug-pipelined-hybrid-project/actions/runs/8747771673/job/24006749488#step:5:10)c3d61b for Scala Compiler version 3.4.1
[debug] [zinc] The Scala compiler is invoked with:
[debug]     -classpath
[debug]     /home/runner/work/sbt-debug-pipelined-hybrid-project/sbt-debug-pipelined-hybrid-project/target/scala-3.4.1/classes:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.1/scala3-library_3-3.4.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.[12](https://github.com/guersam/sbt-debug-pipelined-hybrid-project/actions/runs/8747771673/job/24006749488#step:5:13)/scala-library-2.13.12.jar
[debug] Compilation failed
[error] (Compile / compileIncremental) Compilation failed
guersam commented 5 months ago

Without compilerOrder setting it compiles but the incremental compilation log is similar.

[debug] [zinc] IncrementalCompile -----------
[debug] IncrementalCompile.incrementalCompile
[debug] previous = Stamps for: 4 products, 2 sources, 1 libraries
[debug] current source = Set(${BASE}/src/main/java/example/JavaClass.java, ${BASE}/src/main/scala/example/Main.scala)
[debug] > initialChanges = InitialChanges(Changes(added = Set(), removed = Set(), changed = Set(), unmodified = ...),Set(${BASE}/target/scala-2.13/classes/example/JavaClass.class, ${BASE}/target/scala-2.13/classes/example/Main$.class, ${BASE}/target/scala-2.13/classes/example/Main$delayedInit$body.class, ${BASE}/target/scala-2.13/classes/example/Main.class),Set(),API Changes: Set())
[debug] 
[debug] Initial source changes:
[debug]         removed: Set()
[debug]         added: Set()
[debug]         modified: Set()
[debug] Invalidated products: Set(${BASE}/target/scala-2.13/classes/example/JavaClass.class, ${BASE}/target/scala-2.13/classes/example/Main$.class, ${BASE}/target/scala-2.13/classes/example/Main$delayedInit$body.class, ${BASE}/target/scala-2.13/classes/example/Main.class)
[debug] External API changes: API Changes: Set()
[debug] Modified binary dependencies: Set()
[debug] Initial directly invalidated classes: Set()
[debug] Sources indirectly invalidated by:
[debug]         product: Set(${BASE}/src/main/java/example/JavaClass.java, ${BASE}/src/main/scala/example/Main.scala)
[debug]         binary dep: Set()
[debug]         external source: Set()
[debug] all 2 sources are invalidated
[debug] Initial set of included nodes: 
[debug] Recompiling all sources: number of invalidated sources > 50.0 percent of all sources
[debug] compilation cycle 1
[info] compiling 1 Scala source and 1 Java source to /home/guersam/playground/sbt-pipeline-debug/sbt-pipelining-debug/target/scala-2.13/classes ...
[debug] Getting org.scala-sbt:compiler-bridge_2.13:1.10.0-RC2:compile for Scala 2.13.11
[debug] [zinc] Running cached compiler 21021528 for Scala compiler version 2.13.11
[debug] [zinc] The Scala compiler is invoked with:
[debug]         -classpath
[debug]         /home/guersam/playground/sbt-pipeline-debug/sbt-pipelining-debug/target/scala-2.13/classes:/home/guersam/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.11/scala-library-2.13.11.jar
[debug] Scala compilation took 0.213943626 s
[debug] done compiling Scala sources
guersam commented 5 months ago

I think I found the cause:

  1. When pipelining is enabled, Zinc triggers Scala compilation regardless of the compile order setting. (source)
  2. Only the scala sources are included for compile because the compiler order is not Mixed. (source)
Friendseeker commented 2 weeks ago

I don't know much about pipelining, but since Eugene added the following comment in 5c86cc8b69081d0845407e4a2da3af871eab2eaf

Turn off compileJava when pipelining is on
Note this requires build tools to call `incrementalCompiler.compileAllJava(in, log)` at the right timing.

I guess this is indeed a build tool problem instead of a zinc problem