search

davidB / scala-maven-plugin

The scala-maven-plugin (previously maven-scala-plugin) is used for compiling/testing/running/documenting scala code in maven.
https://davidb.github.io/scala-maven-plugin/
The Unlicense
560 stars 151 forks source link

java.lang.RuntimeException: rt.jar (class sbt.internal.inc.DummyVirtualFile) is not supported #615

Closed LuciferYang closed 2 years ago

LuciferYang commented 2 years ago

reproduce steps, take spark as an example, use scala-maven-plugin 4.6.2 :

  1. mvn clean install -DskipTests -pl core -am
  2. mvn test -pl core

the error message as follows:

[ERROR] ## Exception when compiling 669 sources to /basedir/spark-mine/core/target/scala-2.12/classes
java.lang.RuntimeException: rt.jar (class sbt.internal.inc.DummyVirtualFile) is not supported
scala.sys.package$.error(package.scala:27)
sbt.internal.inc.Locate$.definesClass(Locate.scala:92)
sbt.internal.inc.Locate.definesClass(Locate.scala)
sbt_inc.SbtIncrementalCompiler$1.definesClass(SbtIncrementalCompiler.java:119)
sbt.internal.inc.Locate$.$anonfun$entry$1(Locate.scala:60)
scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
scala.collection.IterableOnceOps.collectFirst(IterableOnce.scala:1079)
scala.collection.IterableOnceOps.collectFirst$(IterableOnce.scala:1071)
scala.collection.AbstractIterator.collectFirst(Iterator.scala:1288)
sbt.internal.inc.Locate$.$anonfun$entry$2(Locate.scala:67)
sbt.internal.inc.LookupImpl.lookupOnClasspath(LookupImpl.scala:51)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$3(IncrementalCommon.scala:764)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$3$adapted(IncrementalCommon.scala:754)
scala.collection.IterableOnceOps.exists(IterableOnce.scala:591)
scala.collection.IterableOnceOps.exists$(IterableOnce.scala:588)
scala.collection.AbstractIterable.exists(Iterable.scala:919)
sbt.internal.inc.IncrementalCommon$.isLibraryChanged$1(IncrementalCommon.scala:754)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$1(IncrementalCommon.scala:774)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$1$adapted(IncrementalCommon.scala:732)
scala.collection.parallel.AugmentedIterableIterator.filter2combiner(RemainsIterator.scala:136)
scala.collection.parallel.AugmentedIterableIterator.filter2combiner$(RemainsIterator.scala:133)
scala.collection.parallel.immutable.ParVector$ParVectorIterator.filter2combiner(ParVector.scala:72)
scala.collection.parallel.ParIterableLike$Filter.leaf(ParIterableLike.scala:1083)
scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:52)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:97)
scala.collection.parallel.Task.tryLeaf(Tasks.scala:55)
scala.collection.parallel.Task.tryLeaf$(Tasks.scala:49)
scala.collection.parallel.ParIterableLike$Filter.tryLeaf(ParIterableLike.scala:1079)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:159)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:156)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:303)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:149)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:148)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:303)
java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:242)
scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:242)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:303)
scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:286)
scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:279)
scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:59)
scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult(Tasks.scala:409)
scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult$(Tasks.scala:409)
scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:75)
scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:932)
scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:52)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:97)
scala.collection.parallel.Task.tryLeaf(Tasks.scala:55)
scala.collection.parallel.Task.tryLeaf$(Tasks.scala:49)
scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:927)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:152)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:148)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:303)
java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

but mvn clean test -pl core is fine

slandelle commented 2 years ago

Hello,

Sorry, but the spark build is way too complex so I can investigate (kind reminder: open-source, free personal time). Please consider building a minimal-reproducible-example.

Otherwise, please consider investigating yourself and contributing a fix.

LuciferYang commented 2 years ago

Hello,

Sorry, but the spark build is way too complex so I can investigate (kind reminder: open-source, free personal time). Please consider building a minimal-reproducible-example.

Otherwise, please consider investigating yourself and contributing a fix.

OK, let me see how to simplify the reproduction steps first

LuciferYang commented 2 years ago

Here is a small project, which only contains a java file, a scala file, a test file and pom xml:

https://github.com/LuciferYang/scala-maven-plugin-test

I test it with java 1.8.0_322 and maven 3.8.4 as follows:

with scala-maven-plugin 4.6.2:

mvn clean install -DskipTests -Dscala-maven-plugin.version=4.6.2
mvn test -Dscala-maven-plugin.version=4.6.2

The error message of mvn test -Dscala-maven-plugin.version=4.6.2 as follows:

[ERROR] ## Exception when compiling 1 sources to /basedir/scala-maven-plugin-test/target/test-classes
java.lang.RuntimeException: rt.jar (class sbt.internal.inc.DummyVirtualFile) is not supported
scala.sys.package$.error(package.scala:27)
sbt.internal.inc.Locate$.definesClass(Locate.scala:92)
sbt.internal.inc.Locate.definesClass(Locate.scala)
sbt_inc.SbtIncrementalCompiler$1.definesClass(SbtIncrementalCompiler.java:119)
sbt.internal.inc.Locate$.$anonfun$entry$1(Locate.scala:60)
scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
scala.collection.IterableOnceOps.collectFirst(IterableOnce.scala:1079)
scala.collection.IterableOnceOps.collectFirst$(IterableOnce.scala:1071)
scala.collection.AbstractIterator.collectFirst(Iterator.scala:1288)
sbt.internal.inc.Locate$.$anonfun$entry$2(Locate.scala:67)
sbt.internal.inc.LookupImpl.lookupOnClasspath(LookupImpl.scala:51)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$3(IncrementalCommon.scala:764)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$3$adapted(IncrementalCommon.scala:754)
scala.collection.IterableOnceOps.exists(IterableOnce.scala:591)
scala.collection.IterableOnceOps.exists$(IterableOnce.scala:588)
scala.collection.AbstractIterable.exists(Iterable.scala:919)
sbt.internal.inc.IncrementalCommon$.isLibraryChanged$1(IncrementalCommon.scala:754)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$1(IncrementalCommon.scala:774)
sbt.internal.inc.IncrementalCommon$.$anonfun$isLibraryModified$1$adapted(IncrementalCommon.scala:732)
scala.collection.parallel.AugmentedIterableIterator.filter2combiner(RemainsIterator.scala:136)
scala.collection.parallel.AugmentedIterableIterator.filter2combiner$(RemainsIterator.scala:133)
scala.collection.parallel.immutable.ParVector$ParVectorIterator.filter2combiner(ParVector.scala:72)
scala.collection.parallel.ParIterableLike$Filter.leaf(ParIterableLike.scala:1083)
scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:52)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:97)
scala.collection.parallel.Task.tryLeaf(Tasks.scala:55)
scala.collection.parallel.Task.tryLeaf$(Tasks.scala:49)
scala.collection.parallel.ParIterableLike$Filter.tryLeaf(ParIterableLike.scala:1079)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:159)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:156)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:303)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:149)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:148)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:303)
java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:242)
scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:242)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:303)
scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:286)
scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:279)
scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:59)
scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult(Tasks.scala:409)
scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult$(Tasks.scala:409)
scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:75)
scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:932)
scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:52)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:97)
scala.collection.parallel.Task.tryLeaf(Tasks.scala:55)
scala.collection.parallel.Task.tryLeaf$(Tasks.scala:49)
scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:927)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:152)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:148)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:303)
java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

with scala-maven-plugin 4.6.1:

mvn clean install -DskipTests -Dscala-maven-plugin.version=4.6.1
mvn test -Dscala-maven-plugin.version=4.6.1
BUILD SUCCESS

@slandelle Is this conducive to continued investigation?

LuciferYang commented 2 years ago

maybe the issue will occur when there are both java files and scala files in the project

LuciferYang commented 2 years ago

also ping @akshaal

slandelle commented 2 years ago

Hello @eed3si9n and @retronym This one is very weird. Could it be possible there's a bug in MappedFileConverter that makes it not work properly on Java 8 (pre-Jigsaw)? Could you please share your thoughts?

LuciferYang commented 2 years ago

@slandelle Will 4.6.3 be released soon?

slandelle commented 2 years ago

It's released

eed3si9n commented 2 years ago

@slandelle Yea. It's possible we have a bug in Zinc but somehow is masked in sbt. See https://github.com/sbt/sbt/pull/5522

slandelle commented 2 years ago

@eed3si9n Wow, you're absolutely right! Bypassing rt.jar before calling Locate.definesClass indeed fixes the issue! Wouldn't it make sense to move this bypass directly in Locate.definesClass?

slandelle commented 2 years ago

For the record: https://github.com/davidB/scala-maven-plugin/commit/04445b15661c94fc1dac7b4b6ff8119f1de4f768

eed3si9n commented 2 years ago

Wouldn't it make sense to move this bypass directly in Locate.definesClass?

Yea. I opened https://github.com/sbt/zinc/issues/1102 for it.

LuciferYang commented 2 years ago

@slandelle @eed3si9n I found that the relevant fix was revert in version 1.7.0. Has this issue been fixed in 1.7.1?

slandelle commented 2 years ago

@LuciferYang The original fix was ugly. Instead, in 04445b15661c94fc1dac7b4b6ff8119f1de4f768, I implemented the same logic/workaround as in sbt, that was also later on implemented in gradle. You can check for yourself that the reproducer you provided works fine with scala-maven-plugin 4.7.1.

LuciferYang commented 2 years ago

Thanks @slandelle I have verified it