Problem adding features to cache

[jbzambon]

I've stood up a NCWMS2 server based on the latest war file and am able to scroll around for a while before running into some errors that cause the data to stop loading. This works if it's the Godiva app trying to access localhost or a remote server supplying WMS calls. After this error pops up, the only solution is to restart the tomcat server. Any advice for what I'm doing wrong?

This is running on a reasonable strong AWS instance (c5.ad-2xlarge) with 8vCPUs and 16gb. I've maxed out the Java RAM to 12GB (setenv.sh attached as setenv.txt). I know WMS performance on AWS isn't as good as dedicated hardware, but I've run into similar issues trying to run a dedicated server off a 36-Xeon core Dell FX2 chassis/blade configuration.

To conserve space, I truncated the catalina.out file to what I believe the problem area is, specifically, this error ERROR uk.ac.rdg.resc.edal.catalogue.DataCatalogue - Problem adding features to cache is repeated about 5,000 times. I checked the content directory and nothing is being written although there is write access to the user.

Besides this problem, any other performance improvement suggestions would be greatly appreciated, thanks!

truncated_catalina.out.txt setenv.txt

[guygriffiths]

I'm not 100% sure what's causing that error, but it's related to caching. However, it shouldn't be a fatal error, since the intended behaviour is to log that message and just not add the feature to the cache. Can you turn off caching and see how it behaves?

In your config.xml ensure you have:

<cache enabled="false">

If that fixes it, great (as a temporary solution while I look into what's going on). If not, hopefully the catalina.out won't be quite so difficult to scan for other error messages which might indicate what's really happening.

Also, what version of Java are you using?

[jbzambon]

Thanks for the help, disabling cache seems to work although seemingly not ideal for load times. I'm considering spinning up a webserver to cache the requests as suggested by @kwilcox through Nginx. Any other performance suggestions are greatly appreciated!

Versions I'm using... /ncwms2/jdk/bin/java --version openjdk 17.0.2 2022-01-18 OpenJDK Runtime Environment (build 17.0.2+8-86) OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing) apache-tomcat-8.5.76 Running ncWMS v2.5.2

Please let me know if you need any other info to troubleshoot, thanks!

[sharon-tickell]

In case it helps anyone else - I have also been running into this issue on the latest NcWMS2 (v2.5.2) with Java 11 as part of the latest Hyrax docker image (opendap/hyrax_ncwms:1.16.8).

If I combine the information in this Stack OverFlow answer about a similar error message and quite a lot of trial and error, it looks like adding the following options to my JAVA_OPTS environment variable may be enough to resolve it:

--add-opens=java.base/java.lang.module=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.math=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.module=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.platform=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.platform.cgroupv1=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED \
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED \
--add-opens=jdk.management.jfr/jdk.management.jfr=ALL-UNNAMED \
--add-opens=jdk.zipfs/jdk.nio.zipfs=ALL-UNNAMED

The trick seems to be to find out the part of the error message that looks like module <first-part> does not "opens <second-part>" to unnamed module, and then add an argument like --add-opens=<first-part>/<second-part>=ALL-UNNAMED. Allowing the first pattern then just means it fails further down on a different one, so repeat until the logs stop complaining about similar issues.

Caveat - I am no kind of a Java expert, so please make your own evaluation of any potential security risk from adding these arguments!

There's some more information on how the --add-opens flag works with recent Java versions at https://docs.oracle.com/en/java/javase/17/migrate/migrating-jdk-8-later-jdk-releases.html#GUID-7744EF96-5899-4FB2-B34E-86D49B2E89B6