How to run test locally in a reproducible way?

[ftomassetti]

I am sorry, but I have no experience with Maven and I am having headaches in getting locally the same results that I get on the CI. For example, at this time on my machine when running mvn test from tool-testuite I get errors I do not get on the CI.

What I am doing is to run:

mvn clean install -e -U -Dmaven.test.skip=true
cd tool-testsuite
mvn test

Am I missing something?

I think that other potential contributors could come from a Gradle background and be running into surprising challenges when dealing with Maven

[KvanTTT]

Could you clarify what errors you have locally?

[ftomassetti]

Sure. The first errors I get are these:

[INFO] --- compiler:3.8.1:testCompile (testCompile) @ antlr4-tool-testsuite ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 52 source files to /Users/ftomassetti/repos/antlr5/tool-testsuite/target/test-classes
[INFO] /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/ToolTestUtils.java: Some input files use or override a deprecated API.
[INFO] /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/ToolTestUtils.java: Recompile with -Xlint:deprecation for details.
[INFO] /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/TestUtils.java: /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/TestUtils.java uses unchecked or unsafe operations.
[INFO] /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/TestUtils.java: Recompile with -Xlint:unchecked for details.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/ToolTestUtils.java:[19,33] package org.antlr.v4.test.runtime does not exist
[ERROR] /Users/ftomassetti/repos/antlr5/tool-testsuite/test/org/antlr/v4/test/tool/ToolTestUtils.java:[20,38] cannot find symbol

The same code ran correctly on the CI

[ftomassetti]

Also, opening that code in IDEA, IDEA has no problem navigating the imports

[KvanTTT]

Have you tried cleaning working directory?

[KvanTTT]

Also, you can try a completely new git clone (or worktree).

[ftomassetti]

I tried both git clean -fdX and mvn clean to no avail

[ftomassetti]

On one of the machines I am using I always get this error:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.antlr.mojo.antlr5.Antlr4MojoTest
[ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 1.006 s <<< FAILURE! -- in org.antlr.mojo.antlr5.Antlr4MojoTest
[ERROR] org.antlr.mojo.antlr5.Antlr4MojoTest.importTokens -- Time elapsed: 0.688 s <<< ERROR!
java.lang.NullPointerException: Cannot invoke "org.apache.maven.model.Plugin.getGroupId()" because the return value of "org.apache.maven.plugin.MojoExecution.getPlugin()" is null
        at org.apache.maven.lifecycle.internal.DefaultMojoExecutionConfigurator.configure(DefaultMojoExecutionConfigurator.java:44)
        at io.takari.maven.testing.Maven331Runtime.lookupConfiguredMojo(Maven331Runtime.java:37)
        at io.takari.maven.testing.Maven325Runtime.executeMojo(Maven325Runtime.java:34)
        at io.takari.maven.testing.TestMavenRuntime.executeMojo(TestMavenRuntime.java:269)
        at org.antlr.mojo.antlr5.Antlr4MojoTest.importTokens(Antlr4MojoTest.java:59)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at io.takari.maven.testing.TestMavenRuntime$6.evaluate(TestMavenRuntime.java:208)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)

Does it ring a bell to someone?

[ftomassetti]

Maybe I am running into this: https://github.com/antlr/antlr4/issues/4512

[ftomassetti]

Running tests in runtime-testsuite instead I get a ton of these errors:

TParser.kt:229:32: error: class 'org.antlr.v5.runtime.kotlin.ParserRuleContext' was compiled with an incompatible version of Kotlin. The binary version of its metadata is 2.0.0, expected version is 1.1.10.
The class is loaded from /Users/federico/.m2/repository/org/antlr/antlr5-kotlin-runtime/0.0.1-SNAPSHOT/antlr5-kotlin-runtime-0.0.1-SNAPSHOT.jar!/org/antlr/v5/runtime/kotlin/ParserRuleContext.class
            _alt = interpreter.adaptivePredict(_input, 2, context)
                               ^

[KvanTTT]

Running tests in runtime-testsuite instead I get a ton of these errors:

Probably using of -Xskip-prerelease-check compiler flag should help. But it's strange that I don't have this error. Moreover, our GitHub CI doesn't have such problems.

Could you try running from the latest dev? I changed compilation scheme there and improved performance a lot.

[ftomassetti]

In general, it seems to me a problem connected with Maven (which may be due to my limited experience with it).

Some issues are having are with remembering to run certain tasks, in a certain order, and with certain flags. For example, I think one first need to install the maven plugin (without running tests) and then to run the tests, but to do that from certain directories.

Other issues have been dependent on the versions of Java or Kotlin used. At some point I was not using the right version of the JVM and the version of Kotlin installed was not the expected one.

Normally all of these things are automatized and checked by gradle builds, so that the process is more reliable and reproducible. I personally think that moving with Gradle will resolve some headaches, but if we do not end up doing that it would be useful to write some guide about compiling the project and running tests, with a troubleshooting section that we grows over time.

[KvanTTT]

In general, it seems to me a problem connected with Maven (which may be due to my limited experience with it).

I suspect the same. We should switch to Gradle ASAP.

[lppedd]

We should switch to Gradle ASAP

Related to this, I did experiment somewhat successfully with antlr-tgen on the Strumenta repository. I've opened some issues there to make it possible to adopt the tool correctly.

Being the grammar and test cases are generated, using the default kotlin.Test framework is possible, and it allows escaping all that test factory and custom runner stuff.

Just my 2c.

[DavidGregory084]

@ftomassetti this first problem you listed (the compilation error) is caused by the current Maven build, which declares a runtime dependency on a published version of the tool module, which is why you have to do mvn -DskipTests install in order to get set up for development in the first place. This means that if you make changes in the tool module, you need to mvn -DskipTests install again or they won't be picked up by the tool-testsuite. We can handle this a lot more gracefully in Gradle by just using project() dependencies.

[ftomassetti]

Thank you @DavidGregory084 . Yes, moving to gradle and using project dependencies would be quite an improvement!