Pro Plugin Development

[sormuras]

I would like to contribute the following two plugins:

• JUnit-Platform-Console-Standalone-Plugin
• Google-Java-Format-Plugin

Both underlying tools are packaged as executable JARs, which can be call via the command line. That's why I think a more general and configurable Download-Dependencies-And-Execute-Plugin would be more useful and be used by other external tools as well. Like checkstyle, coverage, etc.

An usage outline could read like:

set("junit.dependencies", list("junit-platform-console-standalone:1.0.0-SNAPSHOT"))

...

run(... "compiler", "packager", "Download-Dependencies-And-Execute-Plugin(junit)")

Do you think it is worth the effort? Currently, I'm using wget/java commands in the .travis.yml build script.

[forax]

I've tried to not used the command line to run plugins in pro for several reasons. A first reason is speed, pro is currently quite fast (compared to other build tools) because from the VM point of view, pro is one Java program so you do not pay the time to ramp up a new VM each time you run a plugin. A second reason is that pro should offer a programmatic API, hence the name, with useful defaults if your abstraction, is just a Java program or a shell program, you can not provide such API with an implicit configuration. A third reason, is that i want to be able to re-play a build so if you take a look, pro is quite big because it uses its own JDK with its own dependencies, etc. So when you have a version of pro + a specific configuration file should always build the same artifacts whatever is installed .

so i prefer you to develop plugins (a plugin is usually 1 class and 2 interfaces), one plugin can use a jar that comes from Maven Central, call the main method of the main class of that jar, and it will provide a nice experience out of the box, even if it means that it's less flexible.

[sormuras]

I totally agree.

Is there already an example plugin or built-in one that I can use to learn how to write such a plugin that depends on an external jar?

[sormuras]

I started an empty "junitconsole" plugin and need to (auto) fix the JAR in deps/. Locally, I have added deps/junit-platform-console-standalone-1.0.0-20170331.213831-86.jar (soon M4 from Maven Central) which does not contain a module-info.class, yet.

See the skeleton at https://github.com/forax/pro/compare/master...sormuras:plugin_junitconsole

Edit: with https://github.com/junit-team/junit5/issues/761 fixed, the bootstrap module-fixer works like a charm!

[sormuras]

Next task:

[modulefixer] multiple modules [org.junit.jupiter.api, junit.platform.console.standalone@1.0.0-SNAPSHOT] contains package org/junit/jupiter/api/extension
[modulefixer] multiple modules [junit.platform.console.standalone@1.0.0-SNAPSHOT, org.opentest4j] contains package org/opentest4j
[modulefixer] multiple modules [org.junit.jupiter.api, junit.platform.console.standalone@1.0.0-SNAPSHOT] contains package org/junit/jupiter/api
[modulefixer] multiple modules [junit.platform.console.standalone@1.0.0-SNAPSHOT, org.junit.platform.commons] contains package org/junit/platform/commons/annotation
[modulefixer] multiple modules [org.junit.jupiter.api, junit.platform.console.standalone@1.0.0-SNAPSHOT] contains package org/junit/jupiter/api/function
[modulefixer] multiple modules [junit.platform.console.standalone@1.0.0-SNAPSHOT, org.junit.platform.commons] contains package org/junit/platform/commons/meta
[modulefixer] multiple modules [junit.platform.console.standalone@1.0.0-SNAPSHOT, org.junit.platform.commons] contains package org/junit/platform/commons
[modulefixer] multiple modules [junit.platform.console.standalone@1.0.0-SNAPSHOT, org.junit.platform.commons] contains package org/junit/platform/commons/util
[pro] FAILED !        elapsed time 316 ms

How to resolve them?

[forax]

In src/main/java/com.github.forax.pro.plugin.junitconsole/module-info.java, i believe, you do not need to 'opens' com.github.forax.pro.plugin.junitconsole given that you provide your plugin as a com.github.forax.pro.api.Plugin.

The error messages are due to the fact that you have 'split packages', packages that exists in several modules, this is forbidden by Java 9. Only one module can own a package. Otherwise, it means that you can inject any class to any existing package bypassing the security.

There is only one way to solve that, is to move classes between modules to have packages only defined by one module. As a workaround, you can also merge several jars to one bigger jar that will be considered as one module.

The module fixermay solve that for you by merging the jars automatically to create big fat module. But that seems a bit to automagic for me

[sormuras]

Understood.

The split packages result from the "junit-fat-all-standalone.jar" (it contains JUnit 4+5 and dependencies) being visible to the actual test compilation process, which of course refers to the API package required by the test code. Why is the auto-fixed "junit-fat-all-standalone.jar" module visible to user-space (test) code?

[forax]

It's a known limitation of pro, pro creates it's own JDK, and all dependencies are considered as a root modules, if you prefer all dependencies of a plugin is considered as visible as java.base :(

I should try to sandbox each plugin in its own layer or at least, separate the plugins from the rest of the runtime . I will try to come up with a solution for that soon.

[sormuras]

I see. Thanks for the explanation.

Maybe, test execution doesn't have to be part of the pro build. The external tool approach using wget+java works: https://travis-ci.org/sormuras/beethoven/builds/217367445#L345

[sormuras]

Hm, when pro provides (pun intended) all its own dependencies to the projects, how can I configure the resolver/compiler to use the provided modules, like "org.json" or here "junit-platform-console-standalone"?

At the moment, I'm using this "build.pro" file -- but the resolver always tries to reload & fix the junit JAR:

set("compiler.rawArguments", list( "-encoding", "UTF8" ))
run("resolver", "modulefixer", "compiler", "packager", "junitconsole")

Along with a minimal test "module-info.java", requiring exactly the same module as the junitconsole plugin:

module de.sormuras.beethoven {
  requires jdk.compiler;
  requires jdk.scripting.nashorn;
  requires java.desktop;

  // requires org.junit.jupiter.api;
  // requires org.junit.platform.commons;
  // requires org.opentest4j;

  requires junit.platform.console.standalone; // provided by pro plugin "junitconsole"
}

The junitconsole plugin is still a skeleton w/ the fixed JAR included: https://github.com/forax/pro/compare/master...sormuras:plugin_junitconsole

[forax]

The name of a plain old jar (not a modular jar) is extracted from the name of the jar or from the an attribute ModuleName in the jar Manifest by the JDK before being fixed by the module fixer. The module fixer do not change the name of the module.

So apart if your jar is not a modular jar (has a module-info.class), either you can rename the jar (use '.' instead of '-'), or you add the attribute "ModuleName: " with the right name in your jar.

[sormuras]

[...] add the attribute "ModuleName: " with the right name in your jar.

Interesting. Is this attribute an official feature or a good idea implemented in the module fixer?

[forax]

It's an official feature, but i've just checked and it's not yet integrated :(

What is the name of the jar exactly ?

[sormuras]

junit-platform-console-standalone-1.0.0-M4.jar

http://central.maven.org/maven2/org/junit/platform/junit-platform-console-standalone/1.0.0-M4/junit-platform-console-standalone-1.0.0-M4.jar

I renamed it to deps/junit.platform.console.standalone-1.0.0_M4.jar and build pro. But it doesn't show up in the build logs. Like aether and the rest. I guess, all modules from "deps" are merged into target/pro/lib/modules, right?

[sormuras]

Looks like, I got it running now.

I see the log statements like:

...
[compiler] copy src\test\resources to target\test\exploded
[junitconsole] config {
...

}
|  State engine terminated.
|  Restore definitions with: /reload -restore
->

...and the the JShell prompt is blinking. I need to type "/exit" to quit. I guess, I now may start implementing the plugin now.

[sormuras]

Is the initial work here https://github.com/forax/pro/compare/master...sormuras:plugin_junitconsole doing it in the right direction?

[sormuras]

It's an official feature, but i've just checked and it's not yet integrated :(

Can you point me to its documentation?

[sormuras]

The JShell/exit problem was user-made: https://github.com/junit-team/junit5/issues/763 -- calling System.exit(...) in main(...)

Now, the next adventure lies ahead:

|  java.util.ServiceConfigurationError thrown: org.junit.platform.engine.TestEngine: module junit.platform.console.standalone does not declare `uses`
|        at ServiceLoader.fail (ServiceLoader.java:543)
|        at ServiceLoader.checkCaller (ServiceLoader.java:529)
|        at ServiceLoader.<init> (ServiceLoader.java:461)
|        at ServiceLoader.load (ServiceLoader.java:1449)
|        at ServiceLoaderTestEngineRegistry.loadTestEngines (ServiceLoaderTestEngineRegistry.java:29)
|        at LauncherFactory.create (LauncherFactory.java:53)
|        at JunitConsolePlugin.execute (JunitConsolePlugin.java:36)
|        at Pro.execute (Pro.java:267)
|        at Pro.runAll (Pro.java:247)
|        at Pro.run (Pro.java:237)
|        at (#5:1)

Brave New Module World

Is there an option to tell the Module Fixer what services the module uses?

[sormuras]

Maybe, use asm [Class]Remapper to find usages of java.util.ServiceLoader.load(<interface>.class...) and collect all interfaces in a set?

[forax]

yes, good idea, and also provides a way to declare usages in the pro config file.

[sormuras]

(...) the easy way is illegal. (-:

Add TestEngine.class.getModule().addUses(TestEngine.class) to build.pro:

|  java.lang.IllegalCallerException thrown: 
|  unnamed module @47db50c5 != module junit.platform.console.standalone
|        at Module.addUses (Module.java:843)
|        at (#4:1)

[sormuras]

yes, good idea, and also provides a way to declare usages in the pro config file.

Are you working on the topic? If not, I'll prepare a PR.

[forax]

I try to come with something, was a little harder than expected, it seems to work for me.

[sormuras]

oO That really looks good and ... not that simple. The ASM part. Going to re-fix the JAR with your enhanced modulefixer, soon.

[sormuras]

With #15 cherry picked locally, I get:

[pro] registered plugins compiler, convention, junitconsole, linker, modulefixer, packager, resolver, runner, uberpackager
[junitconsole] launcher: org.junit.platform.launcher.core.DefaultLauncher@6eeee66f
[pro] DONE !          elapsed time 7.867 ms

Wohoo! Now, to the real implementation.

[sormuras]

After removing "deps/" and "target/" from my project root, I'm back to FAILED.

run("compiler", "junitconsole")

[pro] registered plugins compiler, convention, junitconsole, linker, modulefixer, packager, resolver, runner, uberpackager
[compiler] test: can not resolve junit.platform.console.standalone from de.sormuras.beethoven -> junit.platform.console.standalone
[pro] FAILED !        elapsed time 2.932 ms

run("resolver", "compiler", "junitconsole")

[pro] registered plugins compiler, convention, junitconsole, linker, modulefixer, packager, resolver, runner, uberpackager
java.lang.IllegalStateException: no value for key dependencies
        at com.github.forax.pro.api@0.9/com.github.forax.pro.api.impl.Configs.lambda$getFromMap$2(Configs.java:113)
        at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1137)
        at com.github.forax.pro.api@0.9/com.github.forax.pro.api.impl.Configs.getFromMap(Configs.java:111)
        at com.github.forax.pro.api@0.9/com.github.forax.pro.api.impl.Configs.lambda$proxy$4(Configs.java:274)
        at com.github.forax.pro.api@0.9/com.github.forax.pro.api.impl.$Proxy10.dependencies(Unknown Source)
        at com.github.forax.pro.plugin.resolver@0.9/com.github.forax.pro.plugin.resolver.ResolverPlugin.execute(ResolverPlugin.java:127)

run("modulefixer", "compiler", "junitconsole")

[pro] registered plugins compiler, convention, junitconsole, linker, modulefixer, packager, resolver, runner, uberpackager
[compiler] test: can not resolve junit.platform.console.standalone from de.sormuras.beethoven -> junit.platform.console.standalone
[pro] FAILED !        elapsed time 2.987 ms

Using the following test module-info.java descriptor:

module de.sormuras.beethoven {
  requires jdk.compiler;
  requires jdk.scripting.nashorn;
  requires java.desktop;

  requires junit.platform.console.standalone; // provided by pro plugin "junitconsole"
}

and this build.pro file:

import static com.github.forax.pro.Pro.*;

//set("pro.loglevel", "debug")
//set("compiler.verbose", true)

set("compiler.rawArguments", list("-encoding", "UTF8"))

// run("modulefixer", "compiler", "junitconsole")
// run("resolver", "compiler", "junitconsole")
run("compiler", "junitconsole")

/exit

[sormuras]

If I remove the requires junit.platform.console.standalone; line from my test descriptor, "pro" reports:

[...]
  (package org.junit.jupiter.api is declared in module junit.platform.console.standalone, but module de.sormuras.beethoven does not read it)
target\test\merged\de.sormuras.beethoven\de\sormuras\beethoven\script\TagTests.java:17: error: static import only from classes and interfaces
import static org.junit.jupiter.api.Assertions.assertEquals;

The module is available. Somewhere.

[sormuras]

Strange. When resolving "junit standalone" again within my project, it works:

test module-info.java

module de.sormuras.beethoven {
  requires jdk.compiler;
  requires jdk.scripting.nashorn;
  requires java.desktop;

  requires junit.standalone;
}

build.pro

set("resolver.dependencies", list(
  "junit.standalone=org.junit.platform:junit-platform-console-standalone:1.0.0-M4"
))
run("resolver", "compiler", "junitconsole")

How can the "junit.platform.console.standalone" module linked into pro and the "junit.standalone" module resolved here live in the same "world"?

If I add the "modulefixer" plugin to "run(...)", I see the split package error messages:

[modulefixer] multiple modules [junit.standalone, junit.platform.console.standalone@1.0.0_M4] contains package org/junit/validator
[modulefixer] multiple modules [junit.standalone, junit.platform.console.standalone@1.0.0_M4] contains package org/junit/jupiter/engine/discovery
[modulefixer] multiple modules [junit.standalone, junit.platform.console.standalone@1.0.0_M4] contains package org/junit/vintage/engine/descriptor
...

[sormuras]

Pro fork with "junitconsole" plugin: https://github.com/forax/pro/compare/master...sormuras:plugin_junitconsole Test-driving "pro" w/ "junitconsole": https://github.com/sormuras/beethoven/blob/java9_and_pro/build.pro

[forax]

So what you are seeing is the bug we already discussed, junitconsole is part of the root module installed with pro, so if you provide another junitconsole, you will have a split package issue because the packages are present twice at runtime.

At the same time, before compiling something, pro checks the configuration, so if your test module does not requires junitconsole, the compiler plugin will report an error.

The best way if think to solve the issue is to sandbox the plugin correctly. I will present 2 talks at DevoxxFR the following days, so i will not be able to do any work on pro before this week end. In the mean time, what you can do as a patch and to de-activate the filter that makes the junitconsole not available when the configuration is checked in the compiler plugin [1]. A line like ModuleFinder systemFinder = ModuleFinder.ofSystem(); at [1] should work as a temporary workaround.

[1] https://github.com/forax/pro/blob/master/src/main/java/com.github.forax.pro.plugin.compiler/com/github/forax/pro/plugin/compiler/CompilerPlugin.java#L172

[forax]

Ok, thinking more about that issue, i now think that the test you be run in their own Java VM, so your plugin should work the same way the runner plugin works and junitconsole should be a dependencies of the project (from the tests of the project) and not a dependency of pro.

[sormuras]

100% d'accord. Closing this issue -- and open new specific ones if necessary.