Jdeps missing dependencies

[Luthien-in-edhil]

Hello,

I'm somewhat reluctant to ask because I have a hunch I'm overlooking something silly, but I feel I've checked everything I could think of a couple of times over. I must confess this is the first time I'm using Java modules, and I have a hunch there is something not quite right in that general direction. This is what I did so far:

• set up the environment according to the readme.md;
• created a new repo using your template;
• cloned it locally, tried it out, got a TestApp.dmg, installed that and it works fine;
• merged the code from my original application in that repository;
• changed some settings, like adding the javafx-fxml and javafx-web modules (I should replace the -web one with something more light-weight, I only use it to render rich text);
• added the required Spring boot, H2 and FXWeaver dependencies and the H2 db files ...

and hit mvn clean install

Result:

There's a small .jar with only the classes from the src/directory, and one of around ~30Mb in the shaded-jar directory. The unpacked-shade dir looks as if it contains what I would expect (eg Spring, H2, Hibernate etc.). But it goes wrong in the jdeps step. The java-tool reports:

Failed to execute goal io.github.wiverson:jtoolprovider-plugin:1.0.25:java-tool (jdeps) on project ithildin: jdeps 1 Details:

[INFO] 
[INFO] --- jtoolprovider-plugin:1.0.25:java-tool (jdeps) @ ithildin ---
[ERROR] jdeps failed with error code [1]
[ERROR]    --add-modules
[ERROR]    javafx.base,javafx.controls,javafx.graphics,javafx.fxml,javafx.web,java.logging
[ERROR]    --generate-module-info
[ERROR]    /Users/luthien/git/aduial/ithildin-app/target/work
[ERROR]    --module-path
[ERROR]    /Users/luthien/git/aduial/ithildin-app/mac-javafx/javafx-sdk-15.0.1/lib/
[ERROR]    /Users/luthien/git/aduial/ithildin-app/target/shaded-jar/ithildin.jar
[INFO] Error: Missing dependencies: classes not found from the module path and classpath.
To suppress this error, use --ignore-missing-deps to continue.
ithildin
   ch.qos.logback.classic.ViewStatusMessagesServlet   -> javax.servlet.http.HttpServletRequest              not found
   ch.qos.logback.classic.ViewStatusMessagesServlet   -> javax.servlet.http.HttpServletResponse             not found
... (4500 more lines like this)

It's a LONG list ;) - it looks like, well, as if I would generate the "Effective POM" in IntelliJ and then spell out every method included therein. Again, I have a hunch this is something really silly, but I can't find it ... 😓

[wiverson]

The good news(?) is that this is pretty much classic modules nightmare stuff. :P

Try tweaking the jdeps module path from:

<modulePath>${javafx.libs}</modulePath>

...to...

<modulePath>${java.home}/jmods/:${project.build.directory}/modules/:${javafx.mods}</modulePath>

...and see if that changes anything.

[wiverson]

As an example that might be helpful, here is a tweaked pom showing the modifications I had to make to add in a JAR that relies on service information (in this case the ikonli icon set): https://gist.github.com/wiverson/fa13d331f5f4a3c35c45bc3bda9be0e4

Key things to look at:

• The maven dependency plugin is used to pull the ikonli artifacts out of the local .m2 repo and put into the target directory
• The shade plugin is told to ignore those artifacts
• jdeps and jpackage add those artifacts to the module path

tbh it's all stupidly complicated, but when it works it's amazing. I'm pretty sure that the answer is to create another Maven plugin that leverages logic like (https://github.com/sandermak/modulescanner) to automatically copy any modules into a folder and generate the proper module paths automatically. In the meantime, it's a lot of Maven hacking to get it to work right.

I just can't believe there isn't better tooling. :P

[Luthien-in-edhil]

Thanks a ton! Can’t wait to try it out, will let you know :)

[Luthien-in-edhil]

I haven't gotten any nearer to overcoming that dependencies issue, but I do understand a lot better now what happens, mostly thanks to your example!

I guess I'm dealing here with the dark side of the Spring Boot magic (why does that remind me of aforementioned shiny jewels ... ) 🙄 The easiest way to include Spring Boot is to add it as a parent in the root pom.xml, which is indeed how I had it set up: dumps all the dependencies in the .m2 maven repository, so that everything needed is available. But it's not straightforward to tell which of that is actually used by the application (though the repackage goal of the spring-boot-maven-plugin seems to know).

The way I use Spring Boot at work is to include it not as a parent pom but like this:

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${springboot.version}</version>
                <scope>import</scope>
                <type>pom</type>
            </dependency>
        </dependencies>
    </dependencyManagement>

Though that results in the identical collection of assorted jars, most of which I won't need. Here's a list of what it contains, causing that astronomical list of missing dependencies. Adding specific dependencies to the maven-dependency-plugin is straightforward enough:

<artifactItem>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter</artifactId>
   <version>${springboot.version}</version>
</artifactItem>
<artifactItem>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
   <version>${springboot.version}</version>
</artifactItem>
<artifactItem>
   <groupId>net.rgielen</groupId>
   <artifactId>javafx-weaver-spring-boot-starter</artifactId>
   <version>${fxweaver.version}</version>
</artifactItem>
<artifactItem>
   <groupId>com.h2database</groupId>
   <artifactId>h2</artifactId>
   <version>1.4.200</version>
</artifactItem>

as is excluding them in the shade plugin configuration:

<exclude>org.springframework.boot:spring-boot-starter</exclude>
<exclude>org.springframework.boot:spring-boot-starter-data-jpa</exclude>
<exclude>net.rgielen:javafx-weaver-spring-boot-starter</exclude>
<exclude>com.h2database:h2</exclude>

(and I changed <modulePath> for jdeps to <modulePath>${java.home}/jmods/:${project.build.directory}/modules/:${javafx.libs}</modulePath> ) - I should find some smart way to capture the path describing all those Spring Boot dependencies - or better still, only the ones that are actually used by the application. Maybe that spring-boot-maven-plugin can do that. When I run that on the previous version of the application, unpack the jar and leave out the javafx jars, there are 46 dependencies left.

I'll try tomorrow to add those to the pom.xml in the way you indicated, and see what that gives 😊

[wiverson]

Hmm.

It might be possible to use https://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html to just drop all of the Maven dependencies into a folder and point module path and/or classpath at that.

A Spring Boot template version of this app has been on my to do list. This is interesting stuff, might be able to put some time in on it next week FWIW. I am hoping for a solution that takes the burden of picking-and-choosing dependencies manually and doing all this config off of the developer. Hmm.

[Luthien-in-edhil]

tbh it's all stupidly complicated, but when it works it's amazing. I'm pretty sure that the answer is to create another Maven plugin that leverages logic like (https://github.com/sandermak/modulescanner) to automatically copy any modules into a folder and generate the proper module paths automatically. In the meantime, it's a lot of Maven hacking to get it to work right.

Hmm well, that might be another thing to look into! It looks indeed like exactly what I am trying to accomplish ...

I just can't believe there isn't better tooling. :P

Absolutely. I have experimented a bit with JavaFX on previous occasions and I cannot remember that it used to be so ridiculously complex. From what I have read and experienced in the past few weeks, this is mostly because of that JavaFX was taken out of the core SDK (though there are some who offer alternate "premixed" SDK's with JavaFX added back in (like Bellsoft - did you have a change to try that?), and of course those modules.

It's odd how I never have run into those modules until I started working with JavaFX again. At work we have been using Java 8 until recently and only started switching to 11 recently. Maybe it doesn't pop up as often in middleware apps - we mostly build Api's that output JSON, Json-LD and such, also using Spring Boot and running in CF. But who knows they're about to make our life there more interesting as well :-)

[wiverson]

Yeah, I'm actually more or less fine with the broad idea of breaking up the JDK into modules - certainly there is a lot of stuff in there that really should go (CORBA? RMI?) and I can see why JavaFX was broken out. Unfortunately, nobody seems to really care about making life easier for dealing with the tooling around modules. Now it's all just confusing and broken and terrible.

I was pretty happy to be able to get working builds just using Maven and the jtoolprovider plugin I wrote - at least it isn't manually managed shell scripts, which is what most folks seem to be using for Java module builds. Even just adding in stuff Ikonli is way too hard - tweaking excludes, etc.

I think a packaging of the JVM with Spring Boot would make for a nice alternative to Electron for JavaScript apps. For most web developers, they could just check out the package, drop their HTML into the right folder and voila! a nice desktop app. Provide a few simple callbacks for all the desktop integration stuff.

Right now I'm sorting out how Windows msi installers work, which is seriously unfun. Working on Maven build stuff seems like a nice break, which... yeesh. ;)

[Luthien-in-edhil]

I was pretty happy to be able to get working builds just using Maven and the jtoolprovider plugin I wrote - at least it isn't manually managed shell scripts, which is what most folks seem to be using for Java module builds.

I think it's quite an achievement already! I'm still sort of reeling of the realisation that what I thought of as simply wrapping the application in colourful paper and tying a ribbon around it turns out to be much more challenging and ground-breaking than building the thing itself ever was 😅 ... this is by now so far removed from the idea of "cross-platform" that using C or Swift seems straightforward in comparison.

[wiverson]

Yeah, I dunno. The results I'm getting are really nice, the build tooling is a mess. Then again, I tried messing around with Electron and https://tauri.studio/en/docs/getting-started/intro/ and... yeah.

I looked at a bunch of different cross-plat dev frameworks a few months ago and realized that they all seemed to be in rougher shape than Java and JavaFX. I'm hoping that a carefully placed Maven plugin might make a lot of this easier. It's not great that the module system seems to be so poorly documented, but then again, I lived through pre-Ant class path on Java... :P

[Luthien-in-edhil]

I wonder, did you ever look into Gluon? I haven't been able to figure out if their setup offers any advantages for desktop apps, because almost everything seems geared towards mobile apps.

Some years ago I briefly looked into Codename One (unclear and the free test version only allows building tiny apps for Android and something called "Windows UWP(?)") and Python + Kivy (FOS, looks good, but it would require refreshing my rusty Python skills).

I suppose you already came across this blog post - it arrives at the same conclusion as you in any case ... so JavaFX it is :)

[Luthien-in-edhil]

Result so far today: I tried this:

It might be possible to use https://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html to just drop all of the Maven dependencies into a folder and point module path and/or classpath at that.

So instead of spelling out the dependencies as in the example with ikonli-javafx you gave here, I tried being Ms Smartypants:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.2</version>
                <executions>
                    <execution>
                        <phase>prepare-package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/module-jars</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

adding ${project.build.directory}/module-jars: to <modulePath> in both the jdeps and jlink configuration. This first revealed some dependency issue with the HikariCP version used in spring-boot-starter and when I excluded that and added the latest version instead, it turned out that HikariCP is a "multi-release" jar; adding <multiRelease>11</multiRelease> to the jdeps configuration seems to have fixed that (unless that is actually causing the following).

This indeed produces a directory filled with 62 jars including all the javafx stuff), but the overall result is that I now have 1000 extra warning lines before the 4500 missing dependencies lines:

Warning: split package: aj.org.objectweb.asm.signature file:///Users/... ... ...
(... 1000 more ...)
Error: Missing dependencies: classes not found from the module path and classpath.
To suppress this error, use --ignore-missing-deps to continue.
ithildin
   ch.qos.logback.classic.ViewStatusMessagesServlet   -> javax.servlet.http.HttpServletRequest              not found
( .... 4500 more ...)

Well, it looked like a good idea 😂 ....

I pushed the pom.xml should you be interested to have a look. I did not yet get around to adding anything to the <excludes> of the shade plugin but I suppose that won't account for the issue.

Next thing I'll try is to do it like your ikonly-javafx example. I also found that Intellij can produce a nice short list of the modules the application is supposed to use, just by clicking Code -> Generate module-info descriptors. This resulted in this list, which surely looks short and sweet compared to everything else:

module ithildin {
    requires java.persistence;
    requires javafx.base;
    requires javafx.controls;
    requires javafx.fxml;
    requires javafx.graphics;
    requires javafx.web;
    requires net.rgielen.fxweaver.core;
    requires spring.beans;
    requires spring.boot;
    requires spring.boot.autoconfigure;
    requires spring.context;
    requires spring.data.commons;
}

I'm still fazed by how modules and jars relate... is it correct that a module can be packaged as a jar?

[wiverson]

AFAIK the IntelliJ module-info that's generated is basically the same thing as an execution of jdeps.

Yes, the most "simple" version of a module is one that is set up by adding an "automatic" entry to a jar manifest. This is appropriate for a simple jar that is just a clump of class files.

The special cases that I know off the top of my head:

• multi release
• services
• native code

Here's an example of how confusing this stuff is: https://stackoverflow.com/a/46514113/311440

The packaging for all of these is confusing and the documentation is poor. At this point it's mostly trial and error, which makes all of this cutting edge stuff!

Whee! ;)

[Luthien-in-edhil]

update: manually added all the dependencies; now the maven clean install fails with:

Failed to execute goal org.moditect:moditect-maven-plugin:1.0.0.RC1:add-module-info (add-module-infos) on project ithildin: Execution add-module-infos of goal org.moditect:moditect-maven-plugin:1.0.0.RC1:add-module-info failed: Couldn't parse /Users/luthien/git/aduial/ithildin-app/target/work/ithildin/module-info.java

Just curious, should this file be already present before the build?

[Luthien-in-edhil]

Ah I guess this explains it:

I suppose that's because of the <multiRelease>11</multiRelease> that I needed to add because of that HikariCP multirelease dependency ...

[wiverson]

Yup, looks like it. Probably can tweak this line to point to it and/or copy it (ugh) if needed:

${project.build.directory}/work/${java-mod-name}/module-info.java

My experience with this stuff has been building a really ugly looking pom that works and then having to go back and clean it up. Sometimes that's what leads to writing a plugin.

[wiverson]

I will note as a fallback that technically jpackage will work without modules - I believe you can just point it at a classpath and have it go. The only real drawback is that then you will include the entire JDK, which I think will probably add about 50mb to your build. You might want to try that angle if the extra 50mb or so isn't a problem.

[Luthien-in-edhil]

mmm ... right now I would gladly add 500mb to the end result if would get me out of this mirror maze! It would of course be great to have a stripped-down module version, but it being 120 Mb instead of 70 is not a show-stopper at all, especially not in this stage. For now I only need a way to distribute Mac and Windows prototypes to a few others so that we can go ahead with further developing the contents and functionality.

I tried a few other things: changing the path to module-info.java leading to errors like this:

[INFO] --- jtoolprovider-plugin:1.0.25:java-tool (jpackage) @ ithildin ---
[DEBUG] Configuring mojo io.github.wiverson:jtoolprovider-plugin:1.0.25:java-tool from plugin realm ClassRealm[plugin>io.github.wiverson:jtoolprovider-plugin:1.0.25, parent: jdk.internal.loader.ClassLoaders$AppClassLoader@73d16e93]
[DEBUG] Configuring mojo 'io.github.wiverson:jtoolprovider-plugin:1.0.25:java-tool' with basic configurator -->
[DEBUG]   (s) args = [@/Users/luthien/git/aduial/ithildin-app/target/packaging/mac-jpackage.txt]
[DEBUG]   (f) echoArguments = false
[DEBUG]   (s) failOnError = true
[DEBUG]   (s) toolName = jpackage
[DEBUG]   (s) writeErrorsToLog = true
[DEBUG]   (s) writeOutputToLog = true
[DEBUG] -- end configuration --
[ERROR] jpackage failed with error code [1]
[ERROR]    @/Users/luthien/git/aduial/ithildin-app/target/packaging/mac-jpackage.txt
[INFO] jdk.incubator.jpackage.internal.PackagerException: jlink failed with: Error: Module net.rgielen.fxweaver.core not found, required by ithildin

This does make some sense though: I'm copying all those dependency jars to target/module-jars (maven-dependency-plugin), then exclude them (maven-shade-plugin) but none of the dependencies are presented to / declared / supplied to jtoolprovider-plugin / jdeps in any way that I can find; so yeah, how can it ever find those?

It looks as if <modules-in-use>javafx.controls,javafx.fxml,javafx.web</modules-in-use> in the properties section of the pom is intended for that - but that is supposed to enumerate modules, not dependency jar files ... ?

I'm a bit ambivalent actually. I would absolutely love to achieve this the hard (module) way. Not so much because of the smaller app size, but it is a great way to become more familiar with the subject as well. I've looked into a few other ways to package the app, but Jpackage (without modules) did not look all that straightforward either ... and besides, the GitHub workflows of this template are fabulous.

[wiverson]

I think one reason the template is blowing up is because it generates the shaded jar by blowing away all of the manifest entries in the declared jars. So, I think the problem with HikariCP is that if you include it in the shaded JAR the removal of the manifest for multi-release gets broken, and if you exclude from the shaded JAR it's not a module independently. The fix (I think) is to generate a module info for HikariCP specifically, making it a module, and then excluding it from the shaded jar. Yeesh.

So, I'm literally working on a Maven plugin right now that loops through all of the declared dependencies in the Maven project, figures out if the dependency is a module or not, and then (hopefully) if it's not a module I can generate the module-info.class via jdeps and attach it. My hope is that this will work for multi-release and make the entire thing just automatic. The strange variety of jar files over the years is kind of crazy - services, multi-release, modules... :P

Crossed-fingers...

[wiverson]

So, a bit of good news, a bit of a head scratcher.

The good news: I added a new command to the jtoolprovider Maven plugin that:

• Looks at all the declared dependencies
• Drops all of the declared dependencies that are already modules into one folder
• Drops all of the declared non-module dependencies into another folder
• Uses jdeps and moditect to automatically turn all of the non-module dependencies into modules
• No longer uses a shaded jar at all, which means stuff like Ikonli should work with no configuration.

I've posted these changes as WIP to the current jtoolprovider plugin. You can check it out, run mvn install and update the version in your project to use the simplified command. I've created a branch on the jpackage-template called copy-dep you can look at to see my WIP testing with the template - and how much smaller the pom is now.

I started testing HikariCP and ran into an interesting problem. Apparently, HikariCP actually IS a module, but for some reason jdeps is insisting on having all of the HikariCP dependencies declared in the HikariCP module-info present, even though virtually all of them are declared as optional. I think the fix for this would be to (sigh) add a command to optionally strip the HikariCP module-info.

I'm declaring victory for the evening, should have more time to look at this later. FWIW I'm tremendously encouraged by how much smaller the pom is for this scenario, and how much more robust it will be going forward. Ironically, apparently there are a LOT of incorrect module-info declarations, which is just...

https://github.com/sormuras/modules/blob/main/doc/suspicious/impostor-modules.md https://github.com/sormuras/modules/tree/main/doc/suspicious

So, in order to deal with incorrect declarations I need to have the ability to quickly mark them as bogus and regenerate anyways. :P

[Luthien-in-edhil]

Sounds interesting! I'll try that as soon as I can tonight :)

[wiverson]

I got it working with HikariCP by adding an option to strip the module-info. It's all working nicely, no need for shade jar anymore either.

I'll publish the plugin update to Maven Central and then update the template. Once all that is done I'll go ahead and add a comment here. Now handles multi-release JARs, option to strip bad module-info from misbehaving modules, etc. Pretty cool. :)

[Luthien-in-edhil]

wow, that's amazing. Congratulations, I am impressed 💃

[wiverson]

FYI, just published the updated plugin to Maven Central and the updated template to this repo as the "copy-dep" branch. I'll be cleaning up this afternoon. I'll post here again when that's done.

Check out how much cleaner the pom is in the new version, lol. Also runs faster too.

[Luthien-in-edhil]

Just had a look 🤩 ... that is quite a difference, yes! Too bad it's nearly 1 am here now, but I will give this a try asap tomorrow!

[wiverson]

KK, looks like everything is working and did a first pass at updating docs. So much nicer now. Let me know how it goes! :)

[Luthien-in-edhil]

I certainly will! 🙂

[Luthien-in-edhil]

Alright, first attempt ... am I right to assume this means jakarta.activation (runtime dep of org.springframework.boot:spring-boot-starter-data-jpa) is one of those badly-behaving dependencies that need to be explicitly declared, like HikariCP?

I'll give that a try in any case.

| => mvn clean install
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< aduial:ithildin >---------------------------
[INFO] Building ithildin 0.2-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ ithildin ---
[INFO] Deleting /Users/luthien/git/ithildin-app/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ ithildin ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources to /Users/luthien/git/ithildin-app/target/packaging
[INFO]
[INFO] --- jtoolprovider-plugin:1.0.30:collect-modules (collect-modules) @ ithildin ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.666 s
[INFO] Finished at: 2021-03-09T16:28:53+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.github.wiverson:jtoolprovider-plugin:1.0.30:collect-modules 
(collect-modules) on project ithildin: Execution collect-modules of goal 
io.github.wiverson:jtoolprovider-plugin:1.0.30:collect-modules failed:
Module jakarta.activation not found, required by java.xml.bind -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

FWIW, I next tried building maven-jpackage-template but that fails rather puzzlingly:

(...)
[INFO] --- jtoolprovider-plugin:1.0.30:java-tool (jpackage) @ maven-jpackage-template ---
[ERROR] No jpackage found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
(...)

Maybe it just needs an IntelliJ reboot ...

PS I can list the contents of the /target directories if that's helpful?

[Luthien-in-edhil]

this is the dependency:tree

[Luthien-in-edhil]

Ahh with the Jakarta added as explicit dependency it now stumbles overlog4j-to-slf4j-2 ... adding that one as well.

[ERROR] Failed to execute goal io.github.wiverson:jtoolprovider-plugin:1.0.30:collect-modules (collect-modules) on project ithildin: Execution collect-modules of goal io.github.wiverson:jtoolprovider-plugin:1.0.30:collect-modules failed: Unable to find a module info for log4j-to-slf4j-2.13.3.jar -> [Help 1]

[Luthien-in-edhil]

Added a few more explicit dependencies; the build seemed to last longer every time suggesting it got more done; but I now came across something that smells a bit like a circular dependency. I added the following dependencies (next to HikariCP that was already in there):

com.sun.activation:jakarta.activation:jar:1.2.2
jakarta.annotation:jakarta.annotation-api:jar:1.3.5
org.apache.logging.log4j:log4j-to-slf4j:jar:2.13.3
org.yaml:snakeyaml:jar:1.27
org.slf4j:slf4j-api:jar:1.7.30

but after adding the last one I again got this warning:

[ERROR] Failed to execute goal io.github.wiverson:jtoolprovider-plugin:1.0.30:collect-modules (collect-modules) on project ithildin: Execution collect-modules of goal io.github.wiverson:jtoolprovider-plugin:1.0.30:collect-modules failed: Unable to find a module info for jakarta.annotation-api-1.3.5.jar -> [Help 1]

... which was already made explicit, so something else is happening here I suppose? If it is any help, I'll push my repository (link) after adding this comment.

[wiverson]

The repo is very helpful.

First, I removed the extra dependencies you added - the goal is for you not to need to do that.

I was able to get pretty far using the configuration below. The debug line spams the console with a lot more information. When I run this, it completes the module-info generations via jdeps, which is absolutely the hard part, but it's failing when trying to attach the module-info to the jar.

For this run, it's failing with the line: Adding info for log4j-to-slf4j-2.13.3.jar

...and looking at the logs, for some reason jdeps is generating the file for this as org.apache.logging.slf4j, so the reason it's failing is just because the jar name doesn't match.

So, this is just a bug in the plugin, where I'm not matching the generated module-info to the jar because sometimes the packages and file names don't match.

Shouldn't be too bad to fix, hopefully can do it today. I'll test against your pom.xml to make sure it's working. This is a very good test case with "jars in the wild" for this plugin. :)

                    <configuration>
                        <providedModuleDirectories>
                            <directory>${javafx.mods}</directory>
                            <directory>${java.home}\jmods\</directory>
                        </providedModuleDirectories>
                        <ignoreJars>
                            <jar>javafx-controls-15.jar</jar>
                            <jar>javafx-fxml-15.jar</jar>
                            <jar>javafx-web-15.jar</jar>
                            <jar>javafx-graphics-15.jar</jar>
                            <jar>javafx-base-15.jar</jar>
                            <jar>jakarta.xml.bind-api</jar>
                            <jar>jaxb-runtime</jar>
                        </ignoreJars>
                        <stripJars>
                            <jar>HikariCP</jar>
                            <jar>classmate</jar>
                            <jar>istack-commons-runtime</jar>
                            <jar>log4j-api</jar>
                            <jar>txw2</jar>
                        </stripJars>
                        <debug>true</debug>
                    </configuration>

[Luthien-in-edhil]

This is so cool! I just tried the debug option to see the magic happening (I also notice all those "Warning: split package:" that I saw some days ago). From what I have read in the past few weeks there are a lot of people out there who are struggling with modules, as is demonstrated by those impostor modules listed in those links you posted. Your plugin, or more broadly, that way of recursively checking all dependencies, and fixing what's broken - that might be a life-saver for many. 😊

[wiverson]

So, I have a working build now I think.

I published an updated version of the plugin, and also made some tweaks to your build.

https://github.com/wiverson/ithildin-app/blob/maven-jpackage-dependency-fix/pom.xml

I had to add the blockhound and cdi-api dependencies to make it happy.

The app just launches and exits right now, no idea why, but at least it's launching.

You may want to tweak your main() method like this to add debug output for compiled binaries:

public static void main(String[] args) {
        try {
            File outputFile = File.createTempFile("debug", ".log", FileSystemView.getFileSystemView().getDefaultDirectory());
            PrintStream output = new PrintStream(new BufferedOutputStream(new FileOutputStream(outputFile)), true);
            System.setOut(output);
            System.setErr(output);
        } catch (IOException e) {
            e.printStackTrace();
        }

        launch(args);
    }

This will cause the app to generate a debug output log when you launch it. Kind of horrible, but may be helpful for debugging issues related to a compiled build. The temp file directory varies by OS, on macOS it drops that in my user directory by default IIRC.

Let me know how it goes!

[wiverson]

kk, I just went ahead and added it because I was curious, and it's a module related error. :P

I'll take a look.

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in Application init method Exception in thread "main" java.lang.RuntimeException: Exception in Application init method at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplication1(Unknown Source) at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication$2(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.IllegalArgumentException: Cannot instantiate interface org.springframework.context.ApplicationContextInitializer : org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer at spring.boot@1/org.springframework.boot.SpringApplication.createSpringFactoriesInstances(Unknown Source) at spring.boot@1/org.springframework.boot.SpringApplication.getSpringFactoriesInstances(Unknown Source) at spring.boot@1/org.springframework.boot.SpringApplication.getSpringFactoriesInstances(Unknown Source) at spring.boot@1/org.springframework.boot.SpringApplication.(Unknown Source) at spring.boot@1/org.springframework.boot.SpringApplication.(Unknown Source) at spring.boot@1/org.springframework.boot.builder.SpringApplicationBuilder.createSpringApplication(Unknown Source) at spring.boot@1/org.springframework.boot.builder.SpringApplicationBuilder.(Unknown Source) at ithildin/aduial.ithildin.application.IthildinJfxApplication.init(Unknown Source) ... 3 more Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer() accessible: module spring.boot.autoconfigure does not "opens org.springframework.boot.autoconfigure" to module spring.core at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source) at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Unknown Source) at java.base/java.lang.reflect.Constructor.setAccessible(Unknown Source) at spring.core@1/org.springframework.util.ReflectionUtils.makeAccessible(Unknown Source) at spring.beans@1/org.springframework.beans.BeanUtils.instantiateClass(Unknown Source) ... 11 more

[Luthien-in-edhil]

I was curious to try, but I don't see any changes? I clean install'ed again to identify those blockhound and cdi-api dependencies but I don't see them anywhere in the output either ....

[wiverson]

So, I wound up trying to just build with jpackage directly and skipping the modules for now. I just sent a PR back with the changes. The good news: the build is much, much faster. The installer is now apx 120MB and the installed app is apx 200MB. 73MB of that is dependencies, a lot of Spring Boot stuff that can be trimmed if you don't need by adding some excludes to the Spring Boot dependency.

Note: I only updated the macOS jpackage command file (mac-jpackage.txt). Either you can update the Windows file (windows-jpackage.txt) yourself or LMK if you need help.

The debug log on launch is now talking about a NullPointer somewhere in setting up the Stage, which is progress I think.

FWIW it looks like Spring Boot, Spring Core etc are doing a bunch of stuff with reflection that seems to be fighting the module system. The stack trace and this error message is particularly flummoxing:

spring.boot.autoconfigure does not "opens org.springframework.boot.autoconfigure" to module spring.core

...as the module-info.java declaration for spring.boot.autoconfigure in that build was explicitly set to be an open module, which according the specs means it should have been open to everything. When I switched jdeps to generate a regular module-info.java, it's exporting that package as well, which should also work but doesn't.

At this point I'm just confused by the whole thing. I don't understand why a module set to be open would generate this error. I Ithink the solution might be to create custom module-info.java files - starting with jdeps generated and then modifying by hand. I've added support to the plugin for exactly this scenario - the plugin can now be set to either generate open, regular, or use a custom module-info.jar for each jar.

Let me know if you have any luck getting it to work with just the regular jpackage bundling up everything on the class path... I think another strategy might be to just use jlink to generate a slimmed down JVM image by directly specifying the JDK modules you actually need and then use jpackage with the build but skip the modules portion. TBH that might be a much easier solution all around.

You can check out the JDK jmods list at $JAVA_HOME/jmods - basically in this scenario you would just specify the modules needed manually and then jlink would build a JVM with just those mods. The good news is that for most apps the list would be pretty short, because most of these would be resolved transitively / automatically. For example, if it's a JavaFX app it might even be as short as "javafx.base,javafx.controls,javafx.graphics,java.logging" and then that would be enough for jlink to make a stripped JVM. That would skip the need for module processing entirely.

Hmm. Very interesting.

As an aside, I'm starting to think that Java modules have really turned into an almost purely JVM internal system. As in, it's clearly very important for the JVM itself, but these errors and problems for building a client-side app are completely ludicrous. Everything I'm doing with this plugin is just a workaround for tooling that should be built in - given a set of jars, jlink and jpackage should be able to generate a JVM without all this craziness. I get a lot of the restrictions for opening/managing libraries that are published, but for a statically compiled end-user desktop app? It makes no sense.

[Luthien-in-edhil]

I'm going to try that now, will report back!

[Luthien-in-edhil]

After some hiccups I got the app to build as well. The first time it failed mysteriously:

[INFO] --- jtoolprovider-plugin:1.0.32:java-tool (jpackage) @ ithildin ---
[1]+  Stopped                 mvn clean install
(ERROR)-(Exit Code 145)-(Unknown error code)
(Thu Mar-3 1:27:31P)-:1)-(luthien:~/git/aduial/ithildin-app)-(78880:      23)
> [ERROR] jpackage failed with error code [1]
[ERROR]    @/Users/luthien/git/aduial/ithildin-app/target/packaging/mac-jpackage.txt
[INFO] java.io.IOException: Command [/usr/bin/hdiutil, attach, /var/folders/6_/w1lyb0ld1zv3qr_cygsbc7c40000gn/T/jdk.incubator.jpackage11973727722991306141/images/ithildin-tmp.dmg, -quiet, -mountroot, /var/folders/6_/w1lyb0ld1zv3qr_cygsbc7c40000gn/T/jdk.incubator.jpackage11973727722991306141/images] exited with 1 code
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
...
[ERROR] Failed to execute goal io.github.wiverson:jtoolprovider-plugin:1.0.32:java-tool (jpackage) on project ithildin: jpackage 1 -> [Help 1]

But after a little shaking ("reload all Maven projects") it changed its mind and produced a .dmg. Which didn't do much. Running it from CMD didn't reveal anything helpful so I rebuilt the app without the debugging just to have it as close to the working (non-jpackage) version as possible. That one did produce some helpful error messages (eg java.lang.NullPointerException: Root cannot be null), and then I tried running it with a Spring Boot run config from IntelliJ. This produced the same error but with more details; I think that shouldn't be too hard to fix, probably some wires got crossed in the past days.

I'm also very much puzzled by how enormously thorny this is. I've only occasionally dabbled with Java client-side applications, starting with Swing which was very straightforward. People were building applets all the time back then. And when I first saw JavaFX I was stunned because it was really beautiful - so much better than the rather clunky "Open Office" look and feel (or Motif or what have you - ah, those memories of SoapUI 😨).

I don't know - maybe this client-side desktop use case of Java is not so much on the radar of those who launch concepts like modules. I have no idea how much it is used compared to the non-gui stuff, but it's probably tiny. But then again - I have built back-end / middleware applications for years and have never even remotely come across something like this. Admittedly this was mostly still in JDK 7 & 8, and only since about a year in 11 - which almost makes me dread what we might be up to when modules start making their way there, too. It would in any case potentially allow for smaller deployments, and as soon as some people start noticing that difference, we might be in for interesting times ...

I'm really sorry that this doesn't yet work given all the effort you put in it, but regardless of the result I appreciate it very much ☺️ ! I need to get some sleep now, I will look further into fixing that NPE tomorrow!

[wiverson]

RE: the time put in, this has all been very helpful research for me! Plus it's made me rethink my overall strategy, which is actually a good thing IMHO.

I think I'm going to pursue / refactor this template with a strategy along the lines of what's described in this article:

https://www.devdungeon.com/content/how-create-java-runtime-images-jlink

In this scenario, the "small JVM" will still be created, and the developer will just get to pick and choose which parts of the JDK will be bundled. The article even includes sample configurations for Swing and JavaFX.

I agree that it's a surprising mess and I have a theory as to what happened...

The JDK had swollen to become an unmanageable mess of circular dependencies internally. Way too many deprecated or useless things had strange dependencies. Modules were created as a way to absolutely enforce breaking the JDK up into pieces as an internal tool. This allowed them to do things like break out JavaFX, begin to increase the pain for continuing to rely on unsupported/deprecated options (e.g. some of the com.sun.unsafe stuff).

They wrote it all up as a spec and got it to work for the JDK itself... and then just kind of dropped the ball on evangelizing and supporting it. I think the management of the JDK team got what it wanted (a way to manage the JVM development) and just didn't have the energy or interest to develop it further.

I'm going to file a bug/feature request with the JDK issue tracker to ask for a combination of either documenting how to get jlink and jdeps to work with Spring Boot and/or providing enhancements for jlink/jdeps to make the process reasonable.

I'm going to directly call out how Spring Boot is the most popular web services tool, and that GraalVM and Spring Native are directly competing in that space.

With regard to desktop development in general, I think the problem is the overall lack of a business model for desktop. It's incredible to me that back in the 1995-97 era, Java was bundled in with every browser and was a direct competitor to Flash. If they had invested in client-side Java back then...

My not-so-secret goal is to make Java desktop development easy and then offer cloud services to make it much easier to work with them - ChangeNode.com will offer automatic updates, crash reporting, and analytics. As part of that effort, I need very simple templates to make it easy to build desktop apps.

I thought that by just writing some Maven plugin stuff to automate the process, it would be possible to add in a few lines of scripts and make an app modular. Now that I've gone through this, I realize that's just not going to happen. Heck, even the build time for running jdeps to process all of the modules is kind of awful.

So, onward and upward. At least I got a pretty solid deep dive into the module system, lol.

[wiverson]

So, good news!

I have rebuilt this template entirely around the idea of skipping the entire "modularize your app" concept and instead switched to "use jlink to build a trimmed custom JVM and just bundle your app with that."

Basically, the template now builds an installer by creating a custom JVM based on the modules you pick (basically just JavaFX modules), collects all of the Maven dependencies as jars and then just builds an ordinary Java app with that. It's much, much more simple.

I've confirmed that this new, refactored template is now working both with the installers and by running javafx:run inside of IntelliJ, so that all looks pretty good.

The hilarious part is that the final generated installer and application sizes are very similar if not basically identical to the sizes of the app I was getting going through all of the stuff with jdeps, etc.

[wiverson]

Ok, I've updated the plugin to play better with Maven (e.g. running install again without running clean first), and also gone through and massively cleaned up, simplified, and documented the pom.xml and the rest of the documentation.

Let me know what you think!

[Luthien-in-edhil]

Ah, I just fixed the NPE (I hadn't set a resource directory with the FXML files in it), and then the application ran fine from IntelliJ again. Then I refactored the pom with all the changes (though probably not the very latest - I pulled it about 2 hrs ago), and then mvn clean install produced the dmg just fine. However ... the packaged app still has another issue that I'll look into in the morning - I have more time during the day tomorrow as well.

This is the first line: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.context.annotation.internalCommonAnnotationProcessor': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.context.annotation.CommonAnnotationBeanPostProcessor]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: javax/naming/NamingException

I pushed the changes and added the stacktrace in case it is of some use to you; but I think it is merely a missing dependency. I did add those dependencies back that you added:

javax.enterprise-cdi-api
io.projectreactor.tools-blockhound
net.bytebuddy-byte-buddy

(what are they for, actually?) but that doesn't cure it just yet ... in any case, I'll try first thing in the morning with the latest changes you pushed :) (and go find where that javax/naming/NamingException class is hiding :P)

A massive thanks again!

[Luthien-in-edhil]

just noticed a broken link in the README.md:

• Can't find jpackage? Probably haven't set up your system to allow Java 15 to enable preview packages.

The URL has brackets in it: ... /blob/main/ ( docs/java-15-jpackage.md )

[Luthien-in-edhil]

Re. the javax/naming/NamingException class not found: it’s in the java.naming package (without the x), but I’m still trying to understand why it cannot be found in the packaged app (I don’t know what .dylib it’s supposed to be in). to be continued ...

(some time later) I can't find it yet. I suppose it might yet be another case of a broken dependency that is otherwise managed by nebulous Spring Boot auto-voodoo. Why on earth this would result in a core java package to be excluded ... beats me. I guess I best leave it for now and go do some chores, maybe that might give me one of those bright flashes of insight that in cartoons are depicted with a light bulb appearing over someone's head 😁

Just thinking out aloud: maybe it would be best to abandon Spring Boot altogether for this sort of thing? It's nice when it works, it makes accessing the database much easier, but what good is that if complicates packaging so much...

[Luthien-in-edhil]

Looked a bit deeper yet, but to no avail. With Spring Boot involved, it looks as if JLink does not look far enough / recurse deeply enough in the dependency tree and hence blips over a transient dependency like that javax/naming/NamingException (if that's what it does indeed) so that it isn't included in the jvm-image directory.

Or I missed something, of course.

Whatever it is, dropping Spring Boot (and FXWeaver) and writing the doa stuff manually is probably a lot less painful. It's just too bad because it looked so tantalizingly close!

I'll give it another try with maven-jpackage-template but without Spring Boot ... having this platform-specific wrapping-up & packaging weighs up against a lot more than just spelling out a few measly dao classes.

[wiverson]

Ok, so I was able to successfully create a working Spring Boot example that uses the Spring Boot / JavaFX weaver project.

It's up on the branch at: https://github.com/wiverson/maven-jpackage-template/tree/spring-boot-test

One very interesting thing - that version of Spring Boot failed when I set the compiler to Java 15 - the version of Spring Boot that's imported by the JavaFX weaver project is using a version of ASM that's too old.

I think that your app may have been failing because of the extra pom.xml dependencies. It may be counter-intuitive, but adding a few of those libraries to the class path causes Spring Boot to find them and try to load them. Try reducing the declared dependencies in the pom and see if that helps.

I thought you were trying to run Spring Boot as an embedded web server. If you are just using it for a few data access classes... hmm. Yeah. There are probably easier/lighter options. Hmm.

[Luthien-in-edhil]

hi, that's intriguing! Great you got at the Spring Boot / JavaCF combination going.

Re. using Spring Boot: it's true that at this stage there are only a few data access classes. At this stage it is just a proof of concept, but if I had known in advance how hairy things would get with it, I wouldn't have bothered writing those manually. But yeah, hindsight has the better eyes. Of course SB offers more than just that. I got interested after reading mr. Gielen's blog post here and I thought that it looked like a pretty good idea. Too bad he doesn't get into packaging the app.

I have already started rewriting the app without Spring Boot ... being able to package the app is more important than DI and JPA autowiring. But I will have a look at those FXWeaver imports in any case; and if that doesn't do the trick then I'll focus on the spring bootless version. Will let you know if I get one or the other working!

[wiverson]

Looks like the javafx-weaver project hasn't been updated in around a year.

Technically it's possible to pick-and-choose bits of Spring. For example, you could just grab the dependency-injection core and Spring Data JPA and embed only those components. That said, my guess is that it's going to be a lot of work sorting out the various components for relatively little reward.

If you are just looking for something to make queries easier, I'd check out MyBatis (https://blog.mybatis.org) and JOOQ instead. (https://www.jooq.org). Both of those are easier for embedding into an app like JavaFX than Spring Boot (IMHO). I love Spring Boot as a web development framework, but my (entirely personal, subjective) take is that it's probably more work to embed in your situation than not.

As an aside, I actually do think there is a scenario for creating a Spring Boot local desktop app, more along the lines of an Electron (or even more precisely, https://neutralino.js.org) app. That would allow a Java/Spring Boot or JavaScript dev to build an app with a Spring Boot backend and ordinary HTML/JavaScript frontend.

[Luthien-in-edhil]

I guess that goes to show I’m way more used to building backend services than desktop gui applications!

Thanks for the suggestions! MyBatis and JOOQ sound like they could be useful, I’ll check those out.

On 16 Mar 2021, at 21:48, Will Iverson @.***> wrote:

 Looks like the javafx-weaver project hasn't been updated in around a year.

Technically it's possible to pick-and-choose bits of Spring. For example, you could just grab the dependency-injection core and Spring Data JPA and embed only those components. That said, my guess is that it's going to be a lot of work sorting out the various components for relatively little reward.

If you are just looking for something to make queries easier, I'd check out MyBatis (https://blog.mybatis.org) and JOOQ instead. (https://www.jooq.org). Both of those are easier for embedding into an app like JavaFX than Spring Boot (IMHO). I love Spring Boot as a web development framework, but my (entirely personal, subjective) take is that it's probably more work to embed in your situation than not.

As an aside, I actually do think there is a scenario for creating a Spring Boot local desktop app, more along the lines of an Electron (or even more precisely, https://neutralino.js.org) app. That would allow a Java/Spring Boot or JavaScript dev to build an app with a Spring Boot backend and ordinary HTML/JavaScript frontend.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[wiverson]

I'm going to go ahead and close this issue down, as I think it's been sorted out - tl;dr don't bother with modules if you don't work on the JDK or JavaFX. Ha.

Please feel free to email me directly if you have any Qs or updates. :) wiverson AT gmail DOT com.