Doesn't play nice with Scala 2.12

[VolkerStolz]

The included dependencies on libraries for Scala 2.11 contain some instances of code that won't run on 2.12, e.g.

$ scala modbat.jar --classpath=modbat-test.jar modbat.test.ChooseTest
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
    at modbat.config.ConfigMgr.<init>(ConfigMgr.scala:73)
    at modbat.mbt.Main$.main(Main.scala:12)
    at modbat.mbt.Main.main(Main.scala)

Unfortunately, it seems that the only solution is to bump the libraries to 2.12 in build.gradle, which may or may not be backwards compatible. It doesn't seem like there's a way of letting Gradle Do The Right Thing automatically. sigh

[cyrille-artho]

You always have to re-compile Scala programs from scratch under the given version, if you want to use a different minor release of Scala. I have not tested Modbat with Scala 2.12, and it probably will not work. There will also be large changes needed to support Java 11 (which Scala itself does not fully support yet).

[clagms]

Maybe my problem is similar. I had no building the code with Gradle 5.5.1. But when running the assembled jar with scala 2.13.0 I get the following error:

PS C:\srcctrl\readonly\modbat\build> scala .\modbat.jar --help
java.lang.ClassNotFoundException: scala.Serializable
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at modbat.mbt.Main$.main(Main.scala:11)
        at modbat.mbt.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:105)
        at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:40)
        at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
        at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:130)
        at scala.reflect.internal.util.ScalaClassLoader.run(ScalaClassLoader.scala:105)
        at scala.reflect.internal.util.ScalaClassLoader.run$(ScalaClassLoader.scala:97)
        at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:130)
        at scala.tools.nsc.CommonRunner.run(ObjectRunner.scala:29)
        at scala.tools.nsc.CommonRunner.run$(ObjectRunner.scala:28)
        at scala.tools.nsc.JarRunner$.run(MainGenericRunner.scala:17)
        at scala.tools.nsc.CommonRunner.runAndCatch(ObjectRunner.scala:35)
        at scala.tools.nsc.CommonRunner.runAndCatch$(ObjectRunner.scala:34)
        at scala.tools.nsc.JarRunner$.runJar(MainGenericRunner.scala:17)
        at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:76)
        at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:91)
        at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:102)
        at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:107)
        at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

As a work around, I've change the build.gradle to produce a fat jar:

jar {
    manifest {
        attributes 'Main-Class': mainClassName,
                   'Name': 'modbat/mbt/',
                   'Specification-Title': 'Modbat: A model-based tester',
                   'Specification-Version': gitDetails.lastTag,
                   'Implementation-Title': 'Modbat'
    }
    from "LICENSE", "README", "CHANGELOG", "asm-LICENSE"
    from configurations.runtimeClasspath.
//                                       findAll { it.name.endsWith('jar') && (it.name.contains('asm') || it.name.contains('config') || it.name.contains('akka')) }.
                                         findAll { it.name.endsWith('jar') }. // Produce fat jar
                                         collect { zipTree(it) }
}

[clagms]

There's another, perhaps more nuanced, problem with scala 2.12: if I build a Model to be verified and compile it in scala 2.12, I get the following:

java -jar lib\modbat.jar --classpath=".\target\scala-2.12\modbat_example_2.12-0.1.jar" SimpleModel
       1      16c71ca7b6c?[2K
[INFO] 0 tests executed, 0 ok, 0 failed.

Where modbat_example_2.12-0.1.jar can be obtained in the sample project I attach: modbat_example.zip

The above command generates a file 16c71ca7b6c.err, whose contents are:

[ERROR] Exception in default (nullary) constructor of main model.
[ERROR] In dot mode, a constructor that ignores any data
[ERROR] is sufficient to visualize the ESFM graph.
[ERROR] java.lang.NoSuchMethodError: modbat.dsl.Model.$init$(Lmodbat/dsl/Model;)V
[ERROR]     at SimpleModel.<init>(SimpleModel.scala:3)
[ERROR]     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[ERROR]     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[ERROR]     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[ERROR]     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)

I was able to ascertain that this is caused by using scala 2.12 because I switched the target scala to 2.11 and it now works as expected.

[wruiwr]

Yeah, the current modbat works with scala 2.11.x.

[cyrille-artho]

Thanks Cláudio, for looking into this. There seems to be a change in how Scala compiles constructors to Java bytecode. One would have to adapt the part that looks up the constructor with reflection to the new naming convention. However, I would rather try to update the code base (in a new branch) directly for Scala 2.13, which is out now. If you are able to help, that would be much appreciated.

[clagms]

I'll be happy to help! I'm not comfortable with the way constructors are looked up though, but I can check if the new branch is working on my side. I might be using this tool for something bigger as well.

[cyrille-artho]

Would you be interested with trying to support Scala 2.13 instead of 2.12? I would rather skip 2.12 as that may double the amount of work to be done and try to support 2.13 instead.

[cyrille-artho]

It looks like scalatest is not yet available for 2.13. Probably also akka-scheduler. So we cannot move to 2.13 for now.

[cyrille-artho]

I've set up a slightly modified build.gradle and test.sh to compile and run Modbat under 2.12, and the output is that a lot of tests depend on 2.11:

# count=259 ok=111 failed=148 skipped=0

Most of the affected functionality (or all of it?) is the pretty-printing of transitions. Some error messages are due to classes being looked up that do not seem to exist (anymore?) after compilation, such as

 modbat/test/ConcurrentModel$$Lambda$123/2080166188.class

I'll have to look into the code that parses the bytecode to get the extra information out; perhaps some of this code is no longer necessary. There is also the branch "unit-test" that does not quite work yet due to strange configuration issues. Help there would be appreciated, as having the unit test framework handle system tests in parallel would speed things up quite a bit...

[cyrille-artho]

I have found the library "sourcecode", which could help us eliminate modbat.util.SourceInfo, which is the biggest problem with Scala 2.12. I will work on this (issue #79) when I have time. After that, porting to 2.12 and later should not be too much work (sourcecode already supports 2.12).

[clagms]

Would you be interested with trying to support Scala 2.13 instead of 2.12? I would rather skip 2.12 as that may double the amount of work to be done and try to support 2.13 instead.

Yes! However, I tried to compile with scala 2.13 (change the dependencies to scala library in the gradle files), and I bump into this issue: https://github.com/gradle/gradle/issues/9857

Most of the affected functionality (or all of it?) is the pretty-printing of transitions.

I've had this error a lot. I corrected some of these errors by unifying how the classes are looked up (essentially getting all the classpath into one single place). See https://github.com/clagms/modbat/commit/30c74a5f94e2c1aa5d1cf0847fde48ba60428385

[cyrille-artho]

Hi, I’m away during the weekend but I’ve looked into this. I’ve made a new branch with work in progress (you can find it via the issue tracker).

Basically, I can change the code to use macros for pretty printing error traces. I hope to do this over the next couple of days/weeks.

However, the model graph for graphviz (dot) is produced statically. It’s not possible to do this at run time.

At least we can probably support the main use cases with macros.

-- Regards, Cyrille Artho

On Aug 17, 2019, at 17:43, Cláudio notifications@github.com wrote:

Would you be interested with trying to support Scala 2.13 instead of 2.12? I would rather skip 2.12 as that may double the amount of work to be done and try to support 2.13 instead.

Yes! However, I tried to compile with scala 2.13 (change the dependencies to scala library in the gradle files), and I bump into this issue: gradle/gradle#9857

Most of the affected functionality (or all of it?) is the pretty-printing of transitions.

I've had this error a lot. I corrected some of these errors by unifying how the classes are looked up (essentially getting all the classpath into one single place). See clagms@30c74a5

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[cyrille-artho]

Branch scala-2.12 now supports compilation and basic functionality on Scala 2.12. Auto-labeling is not supported, as that would require major changes in the bytecode parsing part of SourceInfo. Therefore, many regression tests fail, and the use of auto-labeling will produce a couple of error messages. Other than that, Modbat can be used as normal, in particular, with --no-auto-labels.

[cyrille-artho]

@clagms : Let's wait until we see if the Scala 2.13-related issue with gradle gets resolved. If so, we could try out the current version (from branch scala-2.12) with Scala 2.13 and see if there are any further changes in the bytecode output, which would affect auto-labeling.

[Oscarnorling]

Any news on this front?

Im trying to perform the tutorial and get stuck when I try to run sh compile.sh.

This is the first part of the error I get. error: java.lang.NoClassDefFoundError: scala/Serializable

Seems similar to one of the problems mentioned above...

[cyrille-artho]

I would first try to build Modbat itself with gradle. There are problems with how Scala 2.12 handles the libraries (they are reorganized), so the build scripts in the tutorial cannot find all the libraries. (I got to try this last week on an unusual setup, with similar but different problems). Perhaps it's worth trying to migrate to Gradle 6.1? It seems to have better Scala integration. I'm very happy to include a patch/fix for this but too busy right now to work on this myself.

[cyrille-artho]

@Oscarnorling : I have pushed a few more changes to branch scala 2.12, and there are a few updates. First, the code builds fully now, and many examples work. As far as I can tell, the (many) failing tests are due to changes in how Scala 2.12 and later compiles its lambda expressions. This means that the behavior of many unit tests changes, and their outputs are not compatible. Some examples no longer work because the bytecode parsing that is still used for more user-friendly output messages cannot cope with the new bytecode emitted by Scala 2.12. I'll continue to look into this, and I'm also looking into bringing branch master to Java 11. That seems easier, and may make a transition to Scala 2.12 or 2.13 easier later.