search

scala / bug

Scala 2 bug reports only. Please, no questions — proper bug reports only.
https://scala-lang.org
230 stars 21 forks source link

ScriptEngine fails to init in 2.12.3 with ClassPath #10488

Closed som-snytt closed 6 years ago

som-snytt commented 6 years ago 
$ ~/scala-2.12.2/bin/scala
Welcome to Scala 2.12.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111).
Type in expressions for evaluation. Or try :help.

scala> import javax.script._
import javax.script._

scala> val saved0 = Thread.currentThread.getContextClassLoader
saved0: ClassLoader = scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@3a71c100

scala> val scriptEngineManager = new ScriptEngineManager()
scriptEngineManager: javax.script.ScriptEngineManager = javax.script.ScriptEngineManager@681e144

scala> val se = scriptEngineManager.getEngineByName("scala")
se: javax.script.ScriptEngine = scala.tools.nsc.interpreter.Scripted@1ddc6db2

but

$ scala

     ________ ___   / /  ___  
    / __/ __// _ | / /  / _ | 
  __\ \/ /__/ __ |/ /__/ __ | 
 /____/\___/_/ |_/____/_/ | | 
                          |/  version 2.12.3

scala> import javax.script._
import javax.script._

scala> val saved0 = Thread.currentThread.getContextClassLoader
saved0: ClassLoader = scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@6719a5b8

scala> val scriptEngineManager = new ScriptEngineManager()
scriptEngineManager: javax.script.ScriptEngineManager = javax.script.ScriptEngineManager@3421debd

scala> val se = scriptEngineManager.getEngineByName("scala")
java.lang.AssertionError: assertion failed: Directory file in DirectoryFileLookup cannot be null
  at scala.tools.nsc.classpath.JFileDirectoryLookup.$init$(DirectoryClassPath.scala:120)
  at scala.tools.nsc.classpath.DirectoryClassPath.<init>(DirectoryClassPath.scala:202)
  at scala.tools.nsc.classpath.ClassPathFactory$.newClassPath(ClassPathFactory.scala:78)
  at scala.tools.nsc.classpath.ClassPathFactory.newClassPath(ClassPathFactory.scala:20)
  at scala.tools.nsc.classpath.ClassPathFactory.$anonfun$classesInManifest$1(ClassPathFactory.scala:49)
  at scala.tools.nsc.classpath.ClassPathFactory.classesInManifest(ClassPathFactory.scala:49)
  at scala.tools.util.PathResolver$Calculated$.basis(PathResolver.scala:244)
  at scala.tools.util.PathResolver$Calculated$.containers$lzycompute(PathResolver.scala:248)
  at scala.tools.util.PathResolver$Calculated$.containers(PathResolver.scala:248)
  at scala.tools.util.PathResolver.containers(PathResolver.scala:264)
  at scala.tools.util.PathResolver.computeResult(PathResolver.scala:286)
  at scala.tools.util.PathResolver.result(PathResolver.scala:269)
  at scala.tools.nsc.backend.JavaPlatform.classPath(JavaPlatform.scala:23)
  at scala.tools.nsc.backend.JavaPlatform.classPath$(JavaPlatform.scala:22)
  at scala.tools.nsc.Global$GlobalPlatform.classPath(Global.scala:106)
  at scala.tools.nsc.Global.classPath(Global.scala:117)
  at scala.tools.nsc.Global$GlobalMirror.rootLoader(Global.scala:58)
  at scala.reflect.internal.Mirrors$Roots$RootClass.<init>(Mirrors.scala:276)
  at scala.reflect.internal.Mirrors$Roots.RootClass$lzycompute(Mirrors.scala:290)
  at scala.reflect.internal.Mirrors$Roots.RootClass(Mirrors.scala:290)
  at scala.reflect.internal.Mirrors$Roots$EmptyPackageClass.<init>(Mirrors.scala:299)
  at scala.reflect.internal.Mirrors$Roots.EmptyPackageClass$lzycompute(Mirrors.scala:305)
  at scala.reflect.internal.Mirrors$Roots.EmptyPackageClass(Mirrors.scala:305)
  at scala.reflect.internal.Mirrors$Roots.EmptyPackageClass(Mirrors.scala:245)
  at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:219)
  at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:65)
  at scala.tools.nsc.Global.rootMirror(Global.scala:63)
  at scala.tools.nsc.Global.rootMirror(Global.scala:36)
  at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:267)
  at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:267)
  at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1448)
  at scala.tools.nsc.Global$Run.<init>(Global.scala:1154)
  at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:125)
  at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:147)
  at scala.tools.nsc.interpreter.Scripted.<init>(Scripted.scala:74)
  at scala.tools.nsc.interpreter.Scripted$.apply(Scripted.scala:309)
  at scala.tools.nsc.interpreter.Scripted$Factory.getScriptEngine(Scripted.scala:302)
  at javax.script.ScriptEngineManager.getEngineByName(ScriptEngineManager.java:238)
  ... 29 elided
luben commented 6 years ago

This is not reproducible anymore with Scala-2.12.4

$ scala
Welcome to Scala 2.12.4 (OpenJDK 64-Bit Server VM, Java 1.8.0_151).
Type in expressions for evaluation. Or try :help.

scala> import javax.script._
import javax.script._

scala>  val saved0 = Thread.currentThread.getContextClassLoader
saved0: ClassLoader = scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@74fef3f7

scala>  val scriptEngineManager = new ScriptEngineManager()
scriptEngineManager: javax.script.ScriptEngineManager = javax.script.ScriptEngineManager@6002e944

scala> val se = scriptEngineManager.getEngineByName("scala")
se: javax.script.ScriptEngine = scala.tools.nsc.interpreter.Scripted@29d29657

scala>
som-snytt commented 6 years ago

Maybe fixed per the related ticket. Add a test to backstop, as there are a few moving parts in loading the service.

som-snytt commented 6 years ago

I'll reopen until the test is merged, as (see linked) it will break again by 2.13.

Possibly the class loading bug is why there was never a test for this? That is, the partest failed for a good reason?

vmwabe commented 5 years ago

This issue is still not resolved, if i run the application from the command prompt, it loads fine but once i wrap it using launch4j into an exe, the sripting engine fails to load with the same error. No matter what library i use and i have tried them all including the RC and milestones

SethTisue commented 5 years ago

but once i wrap it using launch4j into an exe

I don't know what the implications of that are, but it sounds like something that could have a lot of implications for the classpath, classloaders, and so on

just because the error message is the same doesn't establish that what you're experiencing is the same bug

som-snytt commented 5 years ago

@vmwabe I'd suggest opening a separate ticket with the conditions of your failure. My workplace just upgraded to Java 11 and a happy Scala script stopped working.

vmwabe commented 5 years ago

This is what i get no matter what library. I would like to start another ticket but since every search leads to this one perhaps it will be beneficial for future victims.

java.lang.AssertionError: assertion failed: Directory file in DirectoryFileLookup cannot be null
    at scala.tools.nsc.classpath.JFileDirectoryLookup.$init$(DirectoryClassPath.scala:120)
    at scala.tools.nsc.classpath.DirectoryClassPath.<init>(DirectoryClassPath.scala:280)
    at scala.tools.nsc.classpath.ClassPathFactory$.newClassPath(ClassPathFactory.scala:78)
    at scala.tools.nsc.classpath.ClassPathFactory.newClassPath(ClassPathFactory.scala:20)
    at scala.tools.nsc.classpath.ClassPathFactory.$anonfun$classesInManifest$1(ClassPathFactory.scala:49)
    at scala.tools.nsc.classpath.ClassPathFactory.classesInManifest(ClassPathFactory.scala:49)
    at scala.tools.util.PathResolver$Calculated$.basis(PathResolver.scala:253)
    at scala.tools.util.PathResolver$Calculated$.containers$lzycompute(PathResolver.scala:257)
    at scala.tools.util.PathResolver$Calculated$.containers(PathResolver.scala:257)
    at scala.tools.util.PathResolver.containers(PathResolver.scala:273)
    at scala.tools.util.PathResolver.computeResult(PathResolver.scala:295)
    at scala.tools.util.PathResolver.result(PathResolver.scala:278)
    at scala.tools.nsc.backend.JavaPlatform.classPath(JavaPlatform.scala:23)
    at scala.tools.nsc.backend.JavaPlatform.classPath$(JavaPlatform.scala:22)
    at scala.tools.nsc.Global$GlobalPlatform.classPath(Global.scala:113)
    at scala.tools.nsc.Global.classPath(Global.scala:124)
    at scala.tools.nsc.Global$GlobalMirror.rootLoader(Global.scala:59)
    at scala.reflect.internal.Mirrors$Roots$RootClass.<init>(Mirrors.scala:274)
    at scala.reflect.internal.Mirrors$Roots.RootClass$lzycompute(Mirrors.scala:288)
    at scala.reflect.internal.Mirrors$Roots.RootClass(Mirrors.scala:288)
    at scala.reflect.internal.Mirrors$Roots$EmptyPackageClass.<init>(Mirrors.scala:297)
    at scala.reflect.internal.Mirrors$Roots.EmptyPackageClass$lzycompute(Mirrors.scala:303)
    at scala.reflect.internal.Mirrors$Roots.EmptyPackageClass(Mirrors.scala:303)
    at scala.reflect.internal.Mirrors$Roots.EmptyPackageClass(Mirrors.scala:243)
    at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:217)
    at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:66)
    at scala.tools.nsc.Global.rootMirror(Global.scala:64)
    at scala.tools.nsc.Global.rootMirror(Global.scala:37)
    at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:286)
    at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:286)
    at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1529)
    at scala.tools.nsc.Global$Run.<init>(Global.scala:1179)
    at scala.tools.nsc.interpreter.IMain.liftedTree1$1(IMain.scala:137)
    at scala.tools.nsc.interpreter.IMain.global$lzycompute(IMain.scala:136)
    at scala.tools.nsc.interpreter.IMain.global(IMain.scala:127)
    at scala.tools.nsc.interpreter.IMain.initializeCompiler(IMain.scala:125)
    at scala.tools.nsc.interpreter.shell.Scripted.<init>(Scripted.scala:30)
    at scala.tools.nsc.interpreter.shell.Scripted$.apply(Scripted.scala:268)
    at scala.tools.nsc.interpreter.shell.Scripted$Factory.getScriptEngine(Scripted.scala:261)
    at javax.script.ScriptEngineManager.getEngineByName(Unknown Source)
    at Blockchain.core.logic.startapps(logic.java:843)
    at Blockchain.core.logic.entrypoint(logic.java:1414)
    at Blockchain.core.logic.init(logic.java:1236)
    at Blockchain.start.main(start.java:160)
vmwabe commented 5 years ago

Perhaps if there was a way to declare what the Directory Classpath is manually I could be able to tweak the error

SethTisue commented 5 years ago

@vmwabe are you on JDK 8, or 11?

vmwabe commented 5 years ago

jdk8

vmwabe commented 5 years ago

i can not develop on jdk10+ yet since it will force my customers to upgrade and too many libraries are broken or no longer in the new jvm. If this can not be resolved then i guess i have to revert to scala 2.7.7 and let the users continue to write scala smart contracts executed using the netgens engine.

SethTisue commented 5 years ago

(8 is the safer choice here. to be clear, I was asking because I thought the problem might be 11-specific, not because I was suggesting you upgrade)

vmwabe commented 5 years ago

I guess I will just be patient until one of the devs can provide a permanent fix.

SethTisue commented 5 years ago

@vmwabe there isn't any permanent “fix” on the way, because we don't yet have a bug report demonstrating that there is actually any bug in Scala itself. perhaps whatever you're doing with launch4j is simply wrong somehow w/r/t classpath, classloaders, etc.

vmwabe commented 5 years ago

There is no bug in scala, just the jsr223 implementation. It works when the big jar is not wrapped but fails when the jar is wrapped. I even tried setting the wrapped jar as a console application instead of a gui to see if it will execute but the same error appeared. The interpreter (IMain) always works smoothly, but the JSR223 Scripted$Factory fails when wrapped. The reason we rely on the jsr223 is because the contracts must be sandboxed to prevent malicious code on our blockchain. All other jsr223 implementations, LuaJ, BSH ,Jython, Groovy, Rhino and Nashorn work well both as interpreters for distributed apps and as ScriptEngines for smart contracts. What I will do is develop my own implemetation of the JSR223 standard for scala although am not well grounded in the language. Wish me luck, i really wish to onboard scala developers in our platform.

SethTisue commented 5 years ago

There is no bug in scala, just the jsr223 implementation

okay. FYI, that implementation was community contributed and is community maintained.

we don't yet have a bug report demonstrating that there is actually any bug in Scala's current JSR-223 implementation. (I informally said "Scala" to refer to "everything in the scala/scala repo", which includes our current JSR-223 implementation.)

What I will do is develop my own implemetation of the JSR223 standard for scala

perhaps the existing one just needs a tweak, or perhaps you need to use it a little differently, and a fresh implementation isn't needed? I don't follow how you're reaching the conclusion that "develop my own" is the right path forward here.

som-snytt commented 5 years ago

From the stack trace, this is failing in the "manifest class path" scan (that was contributed just for JSR 223 IIRC), so it sets "-usemanifestcp". As an experiment or work around, you can try using the apply factory instead of the standard discovery mechanism:

https://github.com/scala/scala/blob/2.13.x/src/repl-frontend/scala/tools/nsc/interpreter/shell/Scripted.scala#L274

The implementation shouldn't blow up if it doesn't find a resource, but what should it do?

som-snytt commented 5 years ago

Perhaps this line to parse the jar URL is failing on a custom URL format, so that ClassPathFactory doesn't detect that it's a jar. Since manifests are filtered for jar protocol, ManifestResources or anything that is a ZipArchive should be confident that isJarOrZip.

Confirming that it doesn't otherwise care what its path is, -Vclasspath:

  DirectoryClassPath(.)
  ManifestResourcesClassPath(bob)
  ManifestResourcesClassPath(bob)
  ManifestResourcesClassPath(bob)
  ManifestResourcesClassPath(bob)
  ManifestResourcesClassPath(bob)
vmwabe commented 5 years ago

Thank you for your resourcefulness. I appreciate it greatly. I think you guys have really nailed the root cause of the problem, it is the manifest. So, is there a java fix or a scala file i can compile and re-inject into the jar ?

vmwabe commented 5 years ago

After recompiling FileUtils based on the ZipArchive hack i get

java.lang.NoSuchMethodError: scala.tools.nsc.classpath.FileUtils$AbstractFileOps$.toURLs$default$1$extension(Lscala/reflect/io/AbstractFile;)Lscala/collection/immutable/Seq;
    at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ManifestResourcesClassPath.$anonfun$asURLs$1(ZipAndJarFileLookupFactory.scala:77)
    at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ManifestResourcesClassPath.asURLs(ZipAndJarFileLookupFactory.scala:77)
    at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$asURLs$1(AggregateClassPath.scala:54)
    at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:106)
    at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:93)
    at scala.collection.immutable.Vector.flatMap(Vector.scala:65)
    at scala.tools.nsc.classpath.AggregateClassPath.asURLs(AggregateClassPath.scala:54)
    at scala.tools.nsc.interpreter.IMain.compilerClasspath(IMain.scala:84)
    at scala.tools.nsc.interpreter.IMain.makeClassLoader(IMain.scala:343)
    at scala.tools.nsc.interpreter.IMain.ensureClassLoader(IMain.scala:266)
    at scala.tools.nsc.interpreter.IMain.classLoader(IMain.scala:269)
    at scala.tools.nsc.interpreter.IMain.runtimeMirror$lzycompute(IMain.scala:159)
    at scala.tools.nsc.interpreter.IMain.runtimeMirror(IMain.scala:159)
    at scala.tools.nsc.interpreter.IMain.$anonfun$getModuleIfDefined$1(IMain.scala:168)
    at scala.tools.nsc.interpreter.IMain.getModuleIfDefined(IMain.scala:161)
    at scala.tools.nsc.interpreter.IMain.readRootPath(IMain.scala:280)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.resolvePathToSymbol(IMain.scala:651)
    at scala.tools.nsc.interpreter.IMain$Request.resultSymbol$lzycompute(IMain.scala:908)
    at scala.tools.nsc.interpreter.IMain$Request.resultSymbol(IMain.scala:907)
    at scala.tools.nsc.interpreter.IMain$Request.$anonfun$typeMap$1(IMain.scala:919)
    at scala.reflect.internal.util.Collections.$anonfun$mapFrom$1(Collections.scala:199)
    at scala.tools.nsc.interpreter.IMain$Request.typeOf$lzycompute(IMain.scala:919)
    at scala.tools.nsc.interpreter.IMain$Request.typeOf(IMain.scala:924)
    at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:886)
    at scala.tools.nsc.interpreter.IMain.$anonfun$compile$1(IMain.scala:487)
    at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:487)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:481)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:464)
    at scala.tools.nsc.interpreter.shell.Scripted.$anonfun$new$1(Scripted.scala:97)
    at scala.tools.nsc.interpreter.shell.ReplReporterImpl.withoutPrintingResults(Reporter.scala:44)
    at scala.tools.nsc.interpreter.shell.Scripted.<init>(Scripted.scala:96)
    at scala.tools.nsc.interpreter.shell.Scripted$.apply(Scripted.scala:268)
    at scala.tools.nsc.interpreter.shell.Scripted$Factory.getScriptEngine(Scripted.scala:261)
    at javax.script.ScriptEngineManager.getEngineByName(Unknown Source)
    at Blockchain.core.logic.startapps(logic.java:787)
    at Blockchain.core.logic.entrypoint(logic.java:1428)
    at Blockchain.core.logic.init(logic.java:1250)
    at Blockchain.start.main(start.java:160)
som-snytt commented 5 years ago

I had a chance to try out launch4j. The manifest URL is just jar:file:/C:/Users/andre/Downloads/app.exe!/META-INF/MANIFEST.MF, so the suffix test would fail.

dwijnand commented 5 years ago

Fixed in https://github.com/scala/scala/pull/8102.

som-snytt commented 4 years ago

ManifestResources

Thank you for your resourcefulness.

Just acknowledging @vmwabe 's pun for the record.

jdimeo commented 3 years ago

I'm getting the exact same error and stack trace on 2,12.13 nightly build 2.12.13-bin-91fa905 but the merged PR seems to indicate it should already have been fixed... is this an issue on my end?

We also observed issues with shading the scala compiler into a shaded jar and have been deploying our code with the assembly plugin (all dependencies in a /lib folder) so the launch4j/manifest issue doesn't apply in my case.

SethTisue commented 3 years ago

https://github.com/scala/scala/pull/8102 fixed it for 2.13.1. We don't routinely backport fixes from 2.13.x to 2.12.x. As far as I can see, a pull request with a backport would be accepted.

jdimeo commented 3 years ago

Thank you for your fast response- I guess I'll look into migrating to 2.13 to get the fixes for this as well as the yet-unreleased back port of the other issue I had (which is why I was on the nightly build thanks to your previous advice).