search

NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a programmer, I want strong modular boundaries (use Java modules introduced by project jigsaw) #208

Open epag opened 2 months ago

epag commented 2 months ago

Author Name: Jesse (Jesse) Original Redmine Issue: 62732, https://vlab.noaa.gov/redmine/issues/62732 Original Date: 2019-04-18

Given a build of WRES When try to use the various modules (jars) in another program or another module Then there should be clear and strong boundaries enforced by the system

Many people worked for a long time to afford this possibility with Java 9 in "Project Jigsaw":https://openjdk.java.net/projects/jigsaw/ so now that we are past Java 8 we can start to make use of these features.

Related issue(s): #102 Redmine related issue(s): 111518, 111532, 119337

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-04-18T16:03:08Z

Rename to not be relative

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-04-18T16:12:13Z

Again clarify title. "Jigsaw" was a project, "Modules" are the construct.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-07-21T20:46:39Z

Should improve startup performance as well.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-09-02T17:38:51Z

Bump. Probably can do it incrementally. Progress in the gradle build in the last few weeks, moving to the @implementation@ and @api@ declarations, which are kind of related to the modular boundaries and descriptions.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-04T16:15:37Z

This should be done regardless of or prior to using native-image or server mode in WRES because this will improve performance in all three situations (the current situation or mode being the third).

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T16:31:56Z

The order in which to do it, from simplest leaf modules with fewest deps up to complex trunk modules with most deps:

  1. wres-messages
  2. wres-worker ...
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:01:56Z

I added this to the @wres-messages@ section of the @build.gradle@:

    // Make this a Java Platform Module System (Java 9 jigsaw) module
    java. modularity.inferModulePath.set( true  )

I added a @wres-messages/src/main/java/module-info.java@ with these contents:

module wres.messages
{
    requires java.base;
    requires com.google.protobuf;
    exports wres.messages.generated;
    exports wres.messages;
}

Then I tried building the jar:

[wres-messages]$ ../gradlew jar

> Task :wres-messages:compileJava FAILED
/home/jesse/code/wres/wres-messages/src/main/java/wres/messages/BrokerHelper.java:21: error: package org.slf4j is not visible
import org.slf4j.Logger;
          ^
  (package org.slf4j is declared in module org.slf4j, but module wres.messages does not read it)
/home/jesse/code/wres/wres-messages/src/main/java/wres/messages/BrokerHelper.java:22: error: package org.slf4j is not visible
import org.slf4j.LoggerFactory;
          ^
  (package org.slf4j is declared in module org.slf4j, but module wres.messages does not read it)
2 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':wres-messages:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 6s
4 actionable tasks: 3 executed, 1 up-to-date

Good. It seems to have had an effect and gradle tried to build it as a module. @wres.messages@ indeed uses @org.slf4j@ so I need to add that to the @module-info.java@.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:03:28Z

After adding @requires org.slf4j;@ to the @module-info.java@, it built the jar:

[wres-messages]$ ../gradlew jar

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5s
6 actionable tasks: 3 executed, 3 up-to-date

Is that it? If so, great!

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:12:28Z

What's inside the jar?

[build]$ cd libs/
[libs]$ ls
wres-messages-20220426-1085a15-dev.jar
[libs]$ unzip wres-messages-20220426-1085a15-dev.jar 
Archive:  wres-messages-20220426-1085a15-dev.jar
   creating: META-INF/
  inflating: META-INF/MANIFEST.MF    
  inflating: module-info.class       
   creating: wres/
   creating: wres/messages/
  inflating: wres/messages/BrokerHelper$Role.class  
  inflating: wres/messages/BrokerHelper.class  
   creating: wres/messages/generated/
  inflating: wres/messages/generated/Job$job$1.class  
  inflating: wres/messages/generated/Job$job$Builder.class  
  inflating: wres/messages/generated/Job$job$Verb$1.class  
  inflating: wres/messages/generated/Job$job$Verb.class  
  inflating: wres/messages/generated/Job$job.class  
  inflating: wres/messages/generated/Job$jobOrBuilder.class  
  inflating: wres/messages/generated/Job.class  
  inflating: wres/messages/generated/JobOutput$job_output$1.class  
  inflating: wres/messages/generated/JobOutput$job_output$Builder.class  
  inflating: wres/messages/generated/JobOutput$job_output.class  
  inflating: wres/messages/generated/JobOutput$job_outputOrBuilder.class  
  inflating: wres/messages/generated/JobOutput.class  
  inflating: wres/messages/generated/JobResult$job_result$1.class  
  inflating: wres/messages/generated/JobResult$job_result$Builder.class  
  inflating: wres/messages/generated/JobResult$job_result.class  
  inflating: wres/messages/generated/JobResult$job_resultOrBuilder.class  
  inflating: wres/messages/generated/JobResult.class  
  inflating: wres/messages/generated/JobStandardStream$job_standard_stream$1.class  
  inflating: wres/messages/generated/JobStandardStream$job_standard_stream$Builder.class  
  inflating: wres/messages/generated/JobStandardStream$job_standard_stream.class  
  inflating: wres/messages/generated/JobStandardStream$job_standard_streamOrBuilder.class  
  inflating: wres/messages/generated/JobStandardStream.class  
  inflating: wres/messages/generated/JobStatus$job_status$1.class  
  inflating: wres/messages/generated/JobStatus$job_status$Builder.class  
  inflating: wres/messages/generated/JobStatus$job_status$Report$1.class  
  inflating: wres/messages/generated/JobStatus$job_status$Report.class  
  inflating: wres/messages/generated/JobStatus$job_status.class  
  inflating: wres/messages/generated/JobStatus$job_statusOrBuilder.class  
  inflating: wres/messages/generated/JobStatus.class  
  inflating: job.proto               
  inflating: job_output.proto        
  inflating: job_result.proto        
  inflating: job_standard_stream.proto  
  inflating: job_status.proto        
  inflating: trustedCertificateAuthorities.jks  
[libs]$ cat META-INF/MANIFEST.MF 
Manifest-Version: 1.0
Implementation-Title: Water Resources Evaluation Service
Implementation-Version: 20220426-1085a15-dev

[libs]$ xxd module-info.class 
0000000: cafe babe 0000 0037 0014 0700 0d01 000a  .......7........
0000010: 536f 7572 6365 4669 6c65 0100 106d 6f64  SourceFile...mod
0000020: 756c 652d 696e 666f 2e6a 6176 6101 0006  ule-info.java...
0000030: 4d6f 6475 6c65 1300 0e13 000f 0100 0731  Module.........1
0000040: 312e 302e 3135 1300 1013 0011 0100 0c32  1.0.15.........2
0000050: 2e30 2e30 2d61 6c70 6861 3714 0012 1400  .0.0-alpha7.....
0000060: 1301 000b 6d6f 6475 6c65 2d69 6e66 6f01  ....module-info.
0000070: 000d 7772 6573 2e6d 6573 7361 6765 7301  ..wres.messages.
0000080: 0009 6a61 7661 2e62 6173 6501 0013 636f  ..java.base...co
0000090: 6d2e 676f 6f67 6c65 2e70 726f 746f 6275  m.google.protobu
00000a0: 6601 0009 6f72 672e 736c 6634 6a01 0017  f...org.slf4j...
00000b0: 7772 6573 2f6d 6573 7361 6765 732f 6765  wres/messages/ge
00000c0: 6e65 7261 7465 6401 000d 7772 6573 2f6d  nerated...wres/m
00000d0: 6573 7361 6765 7380 0000 0100 0000 0000  essages.........
00000e0: 0000 0000 0200 0200 0000 0200 0300 0400  ................
00000f0: 0000 2e00 0500 0000 0000 0300 0600 0000  ................
0000100: 0700 0800 0000 0000 0900 0000 0a00 0200  ................
0000110: 0b00 0000 0000 0c00 0000 0000 0000 0000  ................
0000120: 00                                       .

Nice. Now on to @wres-worker@...

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:23:08Z

Hmm, no such error when building the jar for @wres-worker@ when missing @com.rabbitmq@. But after doing a clean, OK, it had the expected errors.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:30:03Z

Here is the @module-info.java@ for @wres-worker@:

module wres.worker
{
    requires java.base;
    requires com.google.protobuf;
    requires com.rabbitmq.client;
    requires org.slf4j;
    requires wres.messages;
}

It doesn't export anything because it's a top-level application. It only uses this handful of libraries.

What about using @jlink@ now? Can gradle do that for me? Looking...

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:31:20Z

A nice note here at https://docs.gradle.org/6.9.1/userguide/java_library_plugin.html#declaring_module_dependencies

Gradle currently does not automatically check if the dependency declarations are in sync. This may be added in future versions.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:39:31Z

Should @wres-messages@ call @protobuf@ an @api@ dependency? Maybe. Does @wres-worker@ import/use any classes from @protobuf@ that are not @wres.messages@ classes? Yeah, a couple of spots it uses a timestamp and an exception.

Edit: and it already declares protobuf an @api@ dependency, so we need to make the Java Module Directive in @module-info.java@ match the declaration in @build.gradle@, i.e. to be @requires transitive@.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:52:23Z

When making it match, protobuf is reported as an automatic module:

[wres-messages]$ ../gradlew jar

> Task :wres-messages:compileJava
/home/user/code/wres/wres-messages/src/main/java/module-info.java:3: warning: requires transitive directive for an automatic module
    requires transitive com.google.protobuf;
                                  ^
1 warning

BUILD SUCCESSFUL in 4s
6 actionable tasks: 2 executed, 4 up-to-date

Why the warning? https://stackoverflow.com/questions/46750408/why-does-javac-complain-about-named-automatic-modules refers to mailing list answer at https://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-October/013249.html

the main issue is that an automatic module can see classes from the classpath, but it also exports all its package so there is no encapsulation, and once you require one automatic module all automatic modules from the module path are visible. So an automatic module is a great tool when you transitioned to the module world, but in fine, you do not want any automatic modules in you dependency graph.

So we probably do not want @requires transitive@ for protobuf, because that would add everything that protobuf has and uses? I don't quite have the picture in my head but I trust the warning. Going back to @requires@.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:56:43Z

I see this gradle plugin for jlink: https://plugins.gradle.org/plugin/org.beryx.jlink

It warns that it is a complex plugin and to read the docs first. Perhaps I should try jlink by hand first with @wres-worker@ which is a relatively simple application.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T18:58:11Z

Before that, a @javadoc@ task failure when cleaning and compiling the whole tree:

> Task :wres-messages:javadoc
/home/user/code/wres/wres-messages/src/main/java/module-info.java:6: error: package is empty or does not exist: wres.messages.generated
    exports wres.messages.generated;
                         ^
1 error

> Task :wres-messages:javadoc FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':wres-messages:javadoc'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/home/user/code/wres/wres-messages/build/tmp/javadoc/javadoc.options'

Edit: I had previously excluded generated code javadocs:

    javadoc
    {
        // Protobuf does not appear to generate javadoc, so neither will we.
        excludes = ['**/generated/*.java']
    }
</code>

Edit4: removing that exclusion from submodules and from wres-messages resolves the immediate issue but then creates a different issue of annoying warnings from lack of complete javadocs from protoc.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T19:46:15Z

It may be a high bar to use jlink:

[wres-worker]$ find
.
./lib
./lib/conf
./lib/conf/logback.xml
./lib/wres-worker-20220426-1085a15-dev.jar
./lib/wres-messages-20220426-1085a15-dev.jar
./lib/amqp-client-5.14.2.jar
./lib/logback-classic-1.3.0-alpha14.jar
./lib/slf4j-api-2.0.0-alpha7.jar
./lib/protobuf-java-3.20.1.jar
./lib/logback-core-1.3.0-alpha14.jar
./bin
./bin/wres-worker
./bin/wres-worker.bat
[wres-worker]$ $JAVA_HOME/bin/jlink --module-path $JAVA_HOME/jmods:lib --add-modules wres.worker --output workerapp
Error: automatic module cannot be used with jlink: com.rabbitmq.client from file:///home/user/code/wres/wres-worker/build/install/wres-worker/lib/amqp-client-5.14.2.jar
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T20:00:09Z

There are facilities for generating a @module-info.java@ for 3rd-party jars, though. @jdeps@ etc.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T20:01:17Z

Back to that gradle plugin:

Using this Gradle plugin you can create a custom runtime image of your modular application with minimal effort, even if it depends on automatic modules.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T20:19:33Z

I added the plugin to @wres-worker@ and ran the @jlink@ task which on the surface worked fine:

[wres-worker]$ ../gradlew jlink

> Task :wres-worker:jlink
The module name specified in 'application.mainModule' (null) has not the expected value (wres.worker).

BUILD SUCCESSFUL in 17s
13 actionable tasks: 8 executed, 5 up-to-date

However, when running the resulting script:

[image]$ cat bin/wres-worker
#!/bin/sh
SCRIPT_NAME=$(basename "$0")
APP_NAME=${SCRIPT_NAME%.sh}

DIR="${0%/*}"

"$DIR/java" $CDS_JVM_OPTS -Xms64m -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -p "$DIR/../app" -m wres.worker/wres.worker.Worker  "$@"
[image]$ time bin/wres-worker /home/user/code/wres/build/install/wres/bin/wres
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#noProviders for further details.
Exception in thread "main" java.lang.IllegalAccessError: class com.rabbitmq.client.ConnectionFactory (in module wres.merged.module) cannot access class org.slf4j.LoggerFactory (in module org.slf4j) because module wres.merged.module does not read module org.slf4j
    at wres.merged.module@20220426-1085a15-dev/com.rabbitmq.client.ConnectionFactory.<clinit>(ConnectionFactory.java:55)
    at wres.worker/wres.worker.Worker.main(Worker.java:76)

real    0m0.285s
user    0m0.161s
sys 0m0.052s

However, this probably only means we need to add @org.slf4j@ to the merged module. The plugin takes all "automatic" modules and merges them together into a mega-module proper. So perhaps I can have the additional dep added somewhere.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T20:24:48Z

That's a pretty nice sized image, though:

[image]$ du -h
23M ./lib/server
416K    ./lib/security
76K ./lib/jli
94M ./lib
12K ./conf/security/policy/limited
8.0K    ./conf/security/policy/unlimited
24K ./conf/security/policy
88K ./conf/security
104K    ./conf
8.0K    ./include/linux
212K    ./include
40K ./bin
72K ./legal/java.desktop
72K ./legal/java.base
0   ./legal/java.datatransfer
48K ./legal/java.xml
0   ./legal/java.prefs
0   ./legal/java.logging
0   ./legal/java.security.sasl
0   ./legal/java.naming
0   ./legal/java.transaction.xa
0   ./legal/java.sql
0   ./legal/jdk.unsupported
192K    ./legal
95M .

95MiB including java runtime.

Ubi8 is around 216MiB sans java runtime, the @wres-worker@ image is 583MiB including the 2.8MiB of application and by deduction 364MiB of JVM. So an ubi8 image plus this jlink wres-worker image would presumably be around 311MiB instead of 583MiB.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T20:29:47Z

The jlink plugin includes a handy @suggestMergedModuleInfo@ gradle task, which I believe is showing what it generated for the unnamed module subsuming all non-truly-jpms-modules:

[wres-worker]$ ../gradlew suggestMergedModuleInfo

> Task :wres-worker:suggestMergedModuleInfo
mergedModule {
    requires 'java.naming';
    requires 'java.logging';
    requires 'java.security.sasl';
    requires 'java.sql';
    requires 'java.desktop';
    requires 'jdk.unsupported';
}

BUILD SUCCESSFUL in 5s

That's a pretty short list.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T21:02:56Z

Edit: I added this jlink block to the @wres-worker@ block of @build.gradle@ to specify the @org.slf4j@ dependency:

    jlink
    {
        // The merged module is a technique from the plugin that aggregates/merges
        // improper/automatic modules into a proper JPMS module.
        mergedModuleName = 'wres.worker.mergedDepsModule'

        mergedModule
        {
            requires 'java.base'
            requires 'org.slf4j'
        }
    }
</code>

It seems to have partially worked, maybe we need to include logback, though:

[bin]$ time ./wres-worker /home/user/code/wres/build/install/wres/bin/wres
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#noProviders for further details.
Exception in thread "main" java.lang.IllegalStateException: WRES expected to find a file '/wres_secrets/wres-worker_client_private_key_and_x509_cert.p12' with PKCS#12 format, with both a client certificate AND the private key inside, used to authenticate to the broker.
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:313)
    at wres.worker/wres.worker.Worker.main(Worker.java:83)
Caused by: java.io.FileNotFoundException: /wres_secrets/wres-worker_client_private_key_and_x509_cert.p12 (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:306)
    ... 1 more

real    0m0.273s
user    0m0.159s
sys 0m0.086s
[bin]$ pwd
/home/user/code/wres/wres-worker/build/image/bin

That seems to be about twice as fast to get off the ground as this (default script):

[bin]$ cd ../../../
[wres-worker]$ cd build/install/wres-worker/bin/
[bin]$ time ./wres-worker /home/jesse/code/wres/build/install/wres/bin/wres
2022-04-26T16:01:38.536-0500 [main] INFO wres.worker.Worker - Using broker at host 'localhost', vhost 'wres', port '5671'
Exception in thread "main" java.lang.IllegalStateException: WRES expected to find a file '/wres_secrets/wres-worker_client_private_key_and_x509_cert.p12' with PKCS#12 format, with both a client certificate AND the private key inside, used to authenticate to the broker.
    at wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:313)
    at wres.worker.Worker.main(Worker.java:83)
Caused by: java.io.FileNotFoundException: /wres_secrets/wres-worker_client_private_key_and_x509_cert.p12 (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:306)
    ... 1 more

real    0m0.549s
user    0m0.503s
sys 0m0.133s
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T21:12:32Z

Hmm, module paths are different from classpaths, and attempting to add @ requires ch.qos.logback;@ shows an error even in my IDE, same at @jar@ time, same for @ch.qos.logback.classic@ and/or @ch.qos.logback.core@. I have logback-classic specified as a @runtimeOnly@ dependency in gradle but to get it in the module I suppose it has to be compile time.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T21:15:33Z

Something like this, perhaps: https://github.com/beryx/badass-jlink-plugin/issues/191

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T21:23:36Z

Nice. You can use the mergedModule to help include these runtime dependencies in the resulting image. After adding @requiresTransitive 'ch.qos.logback.classic'@ to the @mergedModule@ block in the @jlink@ block exposed by the @org.beryx.jlink@ plugin, it seems logback is there (and increases the runtime a little too):

[bin]$ time ./wres-worker /home/jesse/code/wres/build/install/wres/bin/wres
16:19:51.574 [main] INFO wres.worker.Worker - Using broker at host 'localhost', vhost 'wres', port '5671'
Exception in thread "main" java.lang.IllegalStateException: WRES expected to find a file '/wres_secrets/wres-worker_client_private_key_and_x509_cert.p12' with PKCS#12 format, with both a client certificate AND the private key inside, used to authenticate to the broker.
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:313)
    at wres.worker/wres.worker.Worker.main(Worker.java:83)
Caused by: java.io.FileNotFoundException: /wres_secrets/wres-worker_client_private_key_and_x509_cert.p12 (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:306)
    ... 1 more

real    0m0.337s
user    0m0.294s
sys 0m0.089s

So using the jlink image it only is shaving off about 1/5 of a second from startup. The bigger advantages will be the smaller docker images, better modularity, security.

Edit: and it is still not par because I don't see the logback.xml being applied properly here, so that string formatting could add back another 100ms, who knows?

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T21:30:43Z

Better answer probably here, mentioning that logback uses SPI: https://stackoverflow.com/questions/54777923/logback-in-a-java-9-modular-application-not-working

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-26T21:44:54Z

Or just adding @requires@ works too. I'm not sure that this is the best answer, though:

    jlink
    {
        // The merged module is a technique from the org.beryx.jlink plugin that
        // aggregates/merges improper/automatic modules into a proper JPMS
        // module.
        mergedModuleName = 'wres.worker.mergedDepsModule'

        mergedModule
        {
            requires 'java.base'
            requires 'org.slf4j'
            requires 'ch.qos.logback.classic'
        }
    }

Edit: and on clean build this does not suffice.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T14:31:44Z

Adding @ forceMerge 'slf4j'@ was not enough to bring logback on either.

I tried this, but it causes a different error, which makes sense on second glance because it is forcing inclusion of all dependencies of the declared modules:

    // Jlink is separate from making the jar a module and is at the application
    // level, but made possible by the use of JPMS modules.
    jlink
    {
        // The merged module is a technique from the org.beryx.jlink plugin that
        // aggregates/merges improper/automatic modules into a proper JPMS
        // module.
        mergedModuleName = 'wres.worker.mergedDepsModule'

        forceMerge 'slf4j'
        forceMerge 'logback-classic'
    }
</code>

Error:

[wres-worker]$ ../gradlew clean jar jlink

> Task :wres-worker:compileJava
Note: /home/user/code/wres/wres-worker/src/wres/worker/JobOutputMessenger.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :wres-worker:createMergedModule
Cannot derive uses clause from service loader invocation in: ch/qos/logback/classic/util/ClassicEnvUtil.loadFromServiceLoader().
/home/user/code/wres/wres-worker/build/jlinkbase/tmpjars/wres.worker.mergedDepsModule/module-info.java:49: error: package jakarta.servlet does not exist
    provides jakarta.servlet.ServletContainerInitializer with ch.qos.logback.classic.servlet.LogbackServletContainerInitializer;
                            ^
1 error

> Task :wres-worker:createMergedModule FAILED

FAILURE: Build failed with an exception.

How about add extra dependencies? Like this:

    // Jlink is separate from making the jar a module and is at the application
    // level, but made possible by the use of JPMS modules.
    jlink
    {
        // The merged module is a technique from the org.beryx.jlink plugin that
        // aggregates/merges improper/automatic modules into a proper JPMS
        // module.
        mergedModuleName = 'wres.worker.mergedDepsModule'

        //forceMerge 'slf4j'
        addExtraDependencies 'logback-classic'
    }
</code>

It builds, but logback does not come in, and a different issue occurs:

[wres-worker]$ cd build/image/bin/
[bin]$ time ./wres-worker /home/jesse/code/wres/build/install/wres/bin/wres
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#noProviders for further details.
Exception in thread "main" java.lang.IllegalAccessError: class com.rabbitmq.client.ConnectionFactory (in module wres.worker.mergedDepsModule) cannot access class org.slf4j.LoggerFactory (in module org.slf4j) because module wres.worker.mergedDepsModule does not read module org.slf4j
    at wres.worker.mergedDepsModule@20220426-1085a15-dev/com.rabbitmq.client.ConnectionFactory.<clinit>(ConnectionFactory.java:55)
    at wres.worker/wres.worker.Worker.main(Worker.java:76)

Given the multitude of options available with this plugin I am pretty confident it is a matter of finding the exact right incantation.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T14:37:40Z

Here is the @jlink@ command the plugin generates (found by adding @--debug@ to the @gradlew@ command): @2022-04-27T09:32:57.796-0500 [INFO] [org.gradle.process.internal.DefaultExecHandle] Starting process 'command '/home/user/Downloads/zulu11.56.19-ca-jdk11.0.15-linux_x64/bin/jlink''. Working directory: /home/user/code/wres/wres-worker Command: /home/user/Downloads/zulu11.56.19-ca-jdk11.0.15-linux_x64/bin/jlink -v --module-path /home/user/Downloads/zulu11.56.19-ca-jdk11.0.15-linux_x64/jmods/:/home/user/code/wres/wres-worker/build/jlinkbase/jlinkjars --add-modules wres.worker,wres.worker.mergedDepsModule --output /home/user/code/wres/wres-worker/build/image@

Do we want it added as a module there in @--add-modules@? Perhaps finding the right @jlink@ command first and then figuring out how to get the plugin to do that is better.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T14:51:43Z

I see that the @runtimeClasspath@ is used by default, and I see the logback jars showing up in the jlink jars dir when calling the task that jlink task depends on:

[wres-worker]$ ../gradlew clean jar

> Task :wres-worker:compileJava
Note: /home/jesse/code/wres/wres-worker/src/wres/worker/JobOutputMessenger.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL in 5s
9 actionable tasks: 3 executed, 6 up-to-date
[wres-worker]$ /home/jesse/Downloads/zulu11.56.19-ca-jdk11.0.15-linux_x64/bin/jlink -v --module-path /home/jesse/Downloads/zulu11.56.19-ca-jdk11.0.15-linux_x64/jmods/:/home/jesse/code/wres/wres-worker/build/jlinkbase/jlinkjars --add-modules wres.worker,wres.worker.mergedDepsModule --output /home/jesse/code/wres/^Ces-worker/build/image
[wres-worker]$ ../gradlew prepareModulesDir

BUILD SUCCESSFUL in 10s
12 actionable tasks: 4 executed, 8 up-to-date
[wres-worker]$ find . -name "*.jar"
./build/libs/wres-worker-20220426-1085a15-dev.jar
./build/jlinkbase/jlinkjars/logback-classic-1.3.0-alpha14.jar
./build/jlinkbase/jlinkjars/slf4j-api-2.0.0-alpha7.jar
./build/jlinkbase/jlinkjars/wres-worker.merged.module-20220426-1085a15-dev.jar
./build/jlinkbase/jlinkjars/protobuf-java-3.20.1.jar
./build/jlinkbase/jlinkjars/amqp-client-5.14.2.jar
./build/jlinkbase/jlinkjars/wres-messages-20220426-1085a15-dev.jar
./build/jlinkbase/jlinkjars/logback-core-1.3.0-alpha14.jar
./build/jlinkbase/jlinkjars/wres-worker-20220426-1085a15-dev.jar
./build/jlinkbase/nonmodjars/protobuf-java-3.20.1.jar
./build/jlinkbase/nonmodjars/amqp-client-5.14.2.jar
./build/jlinkbase/tmpmerged/wres-worker.merged.module-20220426-1085a15-dev.jar
./build/jlinkbase/delegating/protobuf-java-3.20.1.jar
./build/jlinkbase/delegating/amqp-client-5.14.2.jar

Running just the jlink command manually after that did not produce the image properly, looking again.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T16:36:37Z

This seems closer to correct, to explicitly add the logback module to the list of app modules:

    // Jlink is separate from making the jar a module and is at the application
    // level, but made possible by the use of JPMS modules.
    jlink
    {
        def myCustomMergedModuleName = 'wres.worker.mergedDepsModule'

        // The merged module is a technique from the org.beryx.jlink plugin that
        // aggregates/merges improper/automatic modules into a proper JPMS
        // module.
        mergedModuleName = myCustomMergedModuleName

        customImage
        {
            appModules = [ 'wres.worker', myCustomMergedModuleName,
                           'ch.qos.logback.classic' ]
        }
    }
</code>

This keeps slf4j and logback out of the merged module, which seems correct because both slf4j and logback are proper modules. Here we see that logback made it (from the lack of SLF4J complaints and also a logback-formatted message) but then a different error:

[wres-worker]$ cd build/image/bin
[bin]$ time ./wres-worker /home/user/code/wres/build/install/wres/bin/wres
11:31:32.323 [main] INFO wres.worker.Worker - Using broker at host 'localhost', vhost 'wres', port '5671'
Exception in thread "main" java.lang.IllegalAccessError: class com.rabbitmq.client.ConnectionFactory (in module wres.worker.mergedDepsModule) cannot access class org.slf4j.LoggerFactory (in module org.slf4j) because module wres.worker.mergedDepsModule does not read module org.slf4j
    at wres.worker.mergedDepsModule@20220426-1085a15-dev/com.rabbitmq.client.ConnectionFactory.<clinit>(ConnectionFactory.java:55)
    at wres.worker/wres.worker.Worker.main(Worker.java:76)

real    0m0.260s
user    0m0.188s
sys 0m0.109s

Maybe we need to add @org.slf4j@ to the 'requires' declaration of the merged module as well.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T16:53:55Z

Logback stays in the @runtimeOnly@ declaration, but is brought into the image with an addition to the @appModules@ list in the @customImage@ block. Slf4j and logback stay as proper JPMS modules and are not included in the merged jar. Classes in the merged jar depend on slf4j, however, and so we explicitly declare that by adding logback to the @requires@ of the @mergedModule@ block.


    // Jlink is separate from making the jar a module and is at the application
    // level, but made possible by the use of JPMS modules.
    jlink
    {
        def myCustomMergedModuleName = 'wres.worker.mergedDepsModule'

        // The merged module is a technique from the org.beryx.jlink plugin that
        // aggregates/merges improper/automatic modules into a proper JPMS
        // module.
        mergedModuleName = myCustomMergedModuleName

        mergedModule
        {
            requires 'java.base'
            requires 'org.slf4j'
        }

        customImage
        {
            appModules = [ 'wres.worker', myCustomMergedModuleName,
                           'ch.qos.logback.classic' ]
        }
    }
</code>

Woot. This seems to have worked. I see logback is present because slf4j complaineth not and I see the logback formatting:

[wres-worker]$ cd build/image/bin
[bin]$ time ./wres-worker /home/jesse/code/wres/build/install/wres/bin/wres
11:45:03.289 [main] INFO wres.worker.Worker - Using broker at host 'localhost', vhost 'wres', port '5671'
Exception in thread "main" java.lang.IllegalStateException: WRES expected to find a file '/wres_secrets/wres-worker_client_private_key_and_x509_cert.p12' with PKCS#12 format, with both a client certificate AND the private key inside, used to authenticate to the broker.
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:313)
    at wres.worker/wres.worker.Worker.main(Worker.java:83)
Caused by: java.io.FileNotFoundException: /wres_secrets/wres-worker_client_private_key_and_x509_cert.p12 (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:306)
    ... 1 more

real    0m0.340s
user    0m0.341s
sys 0m0.039s

Also, the directories and files inside the build directory make some sense:

jlinkbase/jlinkjars
jlinkbase/jlinkjars/slf4j-api-2.0.0-alpha7.jar
jlinkbase/jlinkjars/wres-worker.merged.module-20220426-1085a15-dev.jar
jlinkbase/jlinkjars/protobuf-java-3.20.1.jar
jlinkbase/jlinkjars/amqp-client-5.14.2.jar
jlinkbase/jlinkjars/wres-messages-20220426-1085a15-dev.jar
jlinkbase/jlinkjars/logback-classic-1.3.0-alpha14.jar
jlinkbase/jlinkjars/logback-core-1.3.0-alpha14.jar
jlinkbase/jlinkjars/wres-worker-20220426-1085a15-dev.jar
jlinkbase/nonmodjars
jlinkbase/nonmodjars/protobuf-java-3.20.1.jar
jlinkbase/nonmodjars/amqp-client-5.14.2.jar

In the @jlinkbase/mergedjars@ directory, I only see google and rabbit classes (from protobuf-java and amqp-client respectively), not org.slf4j, not logback. I also have kept the dependencies declaration in @build.gradle@ sound, as it was, such that the primary application build of @wres-worker@ doesn't let logback references sneak into the codebase:

    dependencies
    {
        implementation project( ':wres-messages' )
        implementation 'org.slf4j:slf4j-api:2.0.0-alpha7'
        implementation 'com.rabbitmq:amqp-client:5.14.2'

        runtimeOnly( 'ch.qos.logback:logback-classic:1.3.0-alpha14' )
        {
            // Not used at runtime, bloat
            exclude group: 'edu.washington.cs.types.checker', module: 'checker-framework'
        }

        testImplementation 'junit:junit:4.13.2'
    }
</code>
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T16:57:00Z

Next is to get the @logback.xml@ resource in there. But first, food.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T19:12:26Z

It seems a little janky to copy the @logback.xml@ to the bin directory, but it works:


    // Jlink is separate from making the jar a module and is at the application
    // level, but made possible by the use of JPMS modules.
    jlink
    {
        def myCustomMergedModuleName = 'wres.worker.mergedDepsModule'

        // The merged module is a technique from the org.beryx.jlink plugin that
        // aggregates/merges improper/automatic modules into a proper JPMS
        // module.
        mergedModuleName = myCustomMergedModuleName

        mergedModule
        {
            requires 'java.base'
            requires 'org.slf4j'
        }

        customImage
        {
            appModules = [ 'wres.worker', myCustomMergedModuleName,
                           'ch.qos.logback.classic' ]
        }

        launcher
        {
            jvmArgs = ['-Dlogback.configurationFile=./logback.xml']
        }
    }

    // Copy the logback.xml, other resources into bin. Perhaps we are supposed
    // to include these in the module, but this follows the pattern established
    // above and to-date where we keep these resources outside of jars.
    tasks.jlink.doLast
    {
        copy
        {
            from 'dist/lib/conf'
            into "$imageDir.asFile/bin"
        }
    }
</code>

Showing the directory and output, which has no warning from slf4j, has logback, and found the logback.xml successfully via -D parameter in the launcher script:

[bin]$ ls
java  keytool  logback.xml  wres-worker  wres-worker.bat
[bin]$ time ./wres-worker /home/user/code/wres/build/install/wres/bin/wres
2022-04-27T14:09:23.492-0500 [main] INFO wres.worker.Worker - Using broker at host 'localhost', vhost 'wres', port '5671'
Exception in thread "main" java.lang.IllegalStateException: WRES expected to find a file '/wres_secrets/wres-worker_client_private_key_and_x509_cert.p12' with PKCS#12 format, with both a client certificate AND the private key inside, used to authenticate to the broker.
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:313)
    at wres.worker/wres.worker.Worker.main(Worker.java:83)
Caused by: java.io.FileNotFoundException: /wres_secrets/wres-worker_client_private_key_and_x509_cert.p12 (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
    at wres.messages/wres.messages.BrokerHelper.getSSLContextWithClientCertificate(BrokerHelper.java:306)
    ... 1 more

real    0m0.444s
user    0m0.479s
sys 0m0.083s
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T19:18:46Z

It seems to include much more of the JRE libraries than are necessary, unless protobuf or rabbit or logback depend on them:


[image]$ $JAVA_HOME/bin/jimage list lib/modules 
jimage: lib/modules

Module: ch.qos.logback.classic
    META-INF/MANIFEST.MF
...
    module-info.class

Module: ch.qos.logback.core
    META-INF/MANIFEST.MF
...
    module-info.class

Module: com.google.protobuf
    META-INF/MANIFEST.MF
    module-info.class

Module: com.rabbitmq.client
    META-INF/MANIFEST.MF
    module-info.class

Module: java.base
    META-INF/services/java.nio.file.spi.FileSystemProvider
...
    sun/util/spi/CalendarProvider.class

Module: java.datatransfer
    java/awt/datatransfer/Clipboard.class
...
    sun/datatransfer/resources/flavormap.properties

Module: java.desktop
    com/sun/accessibility/internal/resources/accessibility.class
...
    sun/swing/text/html/FrameEditorPaneTag.class

Module: java.logging
    java/util/logging/ConsoleHandler.class
...
    sun/util/logging/resources/logging_zh_TW.class

Module: java.management
    com/sun/jmx/defaults/JmxProperties.class
...
    sun/management/spi/PlatformMBeanProvider.class

Module: java.naming
    com/sun/jndi/ldap/AbstractLdapNamingEnumeration.class
...
    sun/security/provider/certpath/ldap/LDAPCertStoreImpl.class

Module: java.prefs
    java/util/prefs/AbstractPreferences$1.class
...
    module-info.class

Module: java.security.sasl
    com/sun/security/sasl/ClientFactoryImpl.class
...
    module-info.class

Module: java.sql
    java/sql/Array.class
...
    module-info.class

Module: java.transaction.xa
    javax/transaction/xa/XAException.class
...
    module-info.class

Module: java.xml
    com/sun/java_cup/internal/runtime/Scanner.class
...
    org/xml/sax/helpers/XMLReaderFactory.class

Module: org.slf4j
    META-INF/MANIFEST.MF
...
    org/slf4j/spi/SLF4JServiceProvider.class

Module: wres.messages
    META-INF/MANIFEST.MF
...
    wres/messages/generated/JobStatus.class

Module: wres.worker.mergedDepsModule
    META-INF/MANIFEST.MF
...
    version.properties

Module: wres.worker
    META-INF/MANIFEST.MF
...
    wres/worker/WresProcess.class
[image]$
</PRE>
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T19:28:50Z

Restricting it to @java.base@ in the @customImage@ block, however, goes too far for logback to work, I need more than just @java.base@.

Before:

[image]$ $JAVA_HOME/bin/jimage list lib/modules | grep "^Module"
Module: ch.qos.logback.classic
Module: ch.qos.logback.core
Module: com.google.protobuf
Module: com.rabbitmq.client
Module: java.base
Module: java.datatransfer
Module: java.desktop
Module: java.logging
Module: java.management
Module: java.naming
Module: java.prefs
Module: java.security.sasl
Module: java.sql
Module: java.transaction.xa
Module: java.xml
Module: org.slf4j
Module: wres.messages
Module: wres.worker.mergedDepsModule
Module: wres.worker

The change:

        customImage
        {
            jdkModules = [ 'java.base' ]
            appModules = [ 'wres.worker', myCustomMergedModuleName,
                           'ch.qos.logback.classic' ]
        }
</code>

After:

[image]$ $JAVA_HOME/bin/jimage list lib/modules | grep "^Module"
Module: ch.qos.logback.classic
Module: ch.qos.logback.core
Module: com.google.protobuf
Module: com.rabbitmq.client
Module: java.base
Module: org.slf4j
Module: wres.messages
Module: wres.worker.mergedDepsModule
Module: wres.worker
[image]$ cd bin
[bin]$ time ./wres-worker /home/jesse/code/wres/build/install/wres/bin/wres
Exception in thread "main" java.lang.NoClassDefFoundError: org/xml/sax/InputSource
    at ch.qos.logback.core@1.3.0-alpha14/ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:119)
    at ch.qos.logback.core@1.3.0-alpha14/ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:64)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:68)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:139)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.spi.LogbackServiceProvider.initializeLoggerContext(LogbackServiceProvider.java:50)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.spi.LogbackServiceProvider.initialize(LogbackServiceProvider.java:41)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.bind(LoggerFactory.java:152)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:139)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:421)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:407)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:356)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:382)
    at wres.worker/wres.worker.Worker.<clinit>(Worker.java:30)
Caused by: java.lang.ClassNotFoundException: org.xml.sax.InputSource
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    ... 13 more

real    0m0.241s
user    0m0.213s
sys 0m0.026s

I would have thought adding back @java.xml@ would help find it again, but no:

[bin]$ time ./wres-worker /home/jesse/code/wres/build/install/wres/bin/wres
Exception in thread "main" java.lang.NoClassDefFoundError: org/xml/sax/InputSource
    at ch.qos.logback.core@1.3.0-alpha14/ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:119)
    at ch.qos.logback.core@1.3.0-alpha14/ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:64)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:68)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:139)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.spi.LogbackServiceProvider.initializeLoggerContext(LogbackServiceProvider.java:50)
    at ch.qos.logback.classic@1.3.0-alpha14/ch.qos.logback.classic.spi.LogbackServiceProvider.initialize(LogbackServiceProvider.java:41)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.bind(LoggerFactory.java:152)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:139)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:421)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:407)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:356)
    at org.slf4j@2.0.0-alpha7/org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:382)
    at wres.worker/wres.worker.Worker.<clinit>(Worker.java:30)
Caused by: java.lang.ClassNotFoundException: org.xml.sax.InputSource
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    ... 13 more

real    0m0.245s
user    0m0.201s
sys 0m0.042s
[bin]$ cd ..
[image]$ $JAVA_HOME/bin/jimage list lib/modules | grep "^Module"
Module: ch.qos.logback.classic
Module: ch.qos.logback.core
Module: com.google.protobuf
Module: com.rabbitmq.client
Module: java.base
Module: java.xml
Module: org.slf4j
Module: wres.messages
Module: wres.worker.mergedDepsModule
Module: wres.worker
[image]$ $JAVA_HOME/bin/jimage list lib/modules | grep "org/xml/sax/InputSource"
    jdk/internal/org/xml/sax/InputSource.class
    org/xml/sax/InputSource.class

Weird that logback actually has a version of the same class in there too.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T19:34:15Z

Also weird that my VM just encountered a critical error and needed to be shut down. I guess it is also update day. I will install updates, reboot, and try again.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T21:44:00Z

It seems to be the @java.desktop@ module that is required. That is the last one I would have expected.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T21:53:08Z

Here are modules when declaring @java.base@ and @java.desktop@ as the modules @wres-worker@ depends on:

[image]$ $JAVA_HOME/bin/jimage list lib/modules | grep "^Modu"
Module: ch.qos.logback.classic
Module: ch.qos.logback.core
Module: com.google.protobuf
Module: com.rabbitmq.client
Module: java.base
Module: java.datatransfer
Module: java.desktop
Module: java.prefs
Module: java.xml
Module: org.slf4j
Module: wres.messages
Module: wres.worker.mergedDepsModule
Module: wres.worker

If I remove @java.desktop@ and replace it with @java.xml@ I get @java.lang.ClassNotFoundException: org.xml.sax.InputSource@, which obviously is in @java.xml@, not @java.desktop@. But perhaps it is @java.datatransfer@ and/or @java.prefs@ that it really needs in addition to @java.xml@.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T22:03:04Z

Wow, no it really seems to need @java.desktop@, because I get the @ClassNotFoundException@ with these modules declared: @ jdkModules = [ 'java.base', 'java.xml', 'java.datatransfer', 'java.prefs' ]@

and having these modules present:

[image]$ $JAVA_HOME/bin/jimage list lib/modules | grep "^Modu"
Module: ch.qos.logback.classic
Module: ch.qos.logback.core
Module: com.google.protobuf
Module: com.rabbitmq.client
Module: java.base
Module: java.datatransfer
Module: java.prefs
Module: java.xml
Module: org.slf4j
Module: wres.messages
Module: wres.worker.mergedDepsModule
Module: wres.worker
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T22:09:10Z

When adding the example jlink options, namely @options = ['--strip-debug', '--compress', '2', '--no-header-files', '--no-man-pages']@, the size of the image goes from 95MiB down to 53MiB. That's pretty nice.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T22:32:06Z

This is good enough for the moment. I will push what I have after a local build succeeds.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T22:41:34Z

commit:7c5811597bfac086e6c7248bf0141a123a2194bc has a good start: one library is a JPMS module (wres-messages), one application using that library is a JPMS module (wres-worker), and there is code to generate a runtime image from these modules as well.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T22:48:14Z

Weird build failure when aggregating javadocs:

> Task :aggregateJavadocs
/vlab/jenkins-agents/master-nonpriv/workspace/Verify_OWP_WRES/wres-worker/src/module-info.java:1: error: too many module declarations found
module wres.worker
^

/vlab/jenkins-agents/master-nonpriv/workspace/Verify_OWP_WRES/wres-messages/src/main/java/module-info.java:4: error: module not found: com.google.protobuf
    requires com.google.protobuf;
                       ^
/vlab/jenkins-agents/master-nonpriv/workspace/Verify_OWP_WRES/wres-messages/src/main/java/module-info.java:5: error: module not found: org.slf4j
    requires org.slf4j;
                ^
3 errors

> Task :aggregateJavadocs FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':aggregateJavadocs'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/vlab/jenkins-agents/master-nonpriv/workspace/Verify_OWP_WRES/build/tmp/aggregateJavadocs/javadoc.options'
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T22:53:11Z

Only one module declaration is allowed, and the aggregate javadocs task flattens it all. So perhaps we need to exclude these module declarations from the aggregated javadocs.

I hadn't run that task locally and indeed it reproduces the issue.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T23:44:46Z

Because I could not quickly figure out how to add a working "excludes" to the build script on the javadoc aggregation task and it is getting late, for tonight I disabled the @aggregateJavadocs@ task on the build script. A terrible solution, I know. Perhaps the excludes will be clear and easy in the morning.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T23:47:11Z

One thing I see now is I was modifying the @jacocoRootReport@ task which might have no effect on the @aggregateJavaDocs@ task.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-27T23:56:00Z

https://github.com/nebula-plugins/gradle-aggregate-javadocs-plugin is marked archived by the owner. Well, it still seemed to work.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2022-04-28T22:41:59Z

I tried @io.freefair.aggregate-javadoc@ 6.4.3 but it didn't let me run @./gradlew tasks@.

Then I tried @me.julb.gradleplugins.aggregatejavadoc@ and it failed with the same error as the netflix (one we had earlier). This one lets me exclude whole subprojects but I don't really want to do that either. I would like them all included.