module info for natives have duplicate module names

Version

3.3.2 (nightly)

Platform

Linux x64, Linux arm64, Linux arm32, macOS x64, macOS arm64, Windows x64, Windows x86, Windows arm64

JDK

JDK 9+

Module

All with natives

Bug description

Any jar with natives contains a module info with a module name that is the same as other jars for natives of the same LWJGL module. When loading modules manually using ModuleLayer/ModuleFinder an exception (java.lang.module.FindException) is thrown due to duplicate module names.

These jars should have unique module names, one way to accomplish this would be to put the native platform type at the end of the module name: <module>.natives.arm64

This should not cause any incompatibly with previous releases as these module names are not referenced in user module-info.java.

Stacktrace or crash log output

No response

TLDR: This is by design and shouldn't be an issue in practice as you should never have the native modules for two platforms on the modulepath. You can read about the design decisions here.

Long, biased answer: The way LWJGL handles loading native libraries from Jars is by extracting them into a temporary directory first. This mechanism has failed in the past for unexpected user names (#790) and interfering anti-virus systems (#225) and I (personally) avoid relying on it. Instead, it is generally a good idea to set up your application in a way that the native libraries are loaded from a directory right away. This can, for example, be achieved by installing the native libraries into a bin subdirectory in your application folder and setting Configuration.LIBRARY_PATH (ref) accordingly. Having the natives on the classpath/modulepath is useful during development though. However, your build tool should provide a mechanism for making sure that only the modules for your platform are added to the configuration. (You can find examples of how this can be done in the build customizer.)

This should not cause any incompatibly with previous releases as these module names are not referenced in user module-info.java.

I don't think this statement is true. If you do decide to let LWJGL take care of extracting the native libraries, it should be necessary to require the native modules in your module declaration for them to be visible to LWJGL. Thus, the solution here is to require the native modules (e.g., org.lwjgl.natives, org.lwjgl.glfw.natives) and ensure that only the modules for the desired platform are actually available on the modulepath.

Hello @TheMrMilchmann it is not a matter of loading the native jars into the classpath. If the native jars exist in the same folder as each other, the ModuleFinder looks at all jars in the folder (not just the jar you are trying to load) and will throw the exception. In my case, I am letting LWJGL handle the natives entirely and not even attempting to load a LWJGL jar, but because the LWJGL jars are in the same directory and have the same module names between multiple jars this causes the exception.

It is not always possible to control/prevent having multiple natives in the same directory (module path), and when LWJGL is loading the jars there is no issues. I have many applications that contain all the jars for different platforms and LWJGL always loads the correct natives.

I don't think this statement is true. If you do decide to let LWJGL take care of extracting the native libraries, it should be necessary to require the native modules in your module declaration for them to be visible to LWJGL. Thus, the solution here is to require the native modules (e.g., org.lwjgl.natives, org.lwjgl.glfw.natives) and ensure that only the modules for the desired platform are actually available on the modulepath.

I have never had to add the native module names to my module-info.java. Do you have an example of this?

Hello @TheMrMilchmann it is not a matter of loading the native jars into the classpath. If the native jars exist in the same folder as each other, the ModuleFinder looks at all jars in the folder (not just the jar you are trying to load) and will throw the exception.

There are two ways to circumvent this: You could either add the individual jars (instead of the directory to the modulepath), or move the jars to different directories per platform.

It is not always possible to control/prevent having multiple natives in the same directory (module path), and when LWJGL is loading the jars there is no issues. I have many applications that contain all the jars for different platforms and LWJGL always loads the correct natives.

That was (and should still be) possible when using LWJGL on the classpath but I'd strongly recommend shipping platform-specific bundles of your application instead.

I have never had to add the native module names to my module-info.java. Do you have an example of this?

I wrote a very small test program to quickly validate my claim: https://github.com/TheMrMilchmann/MCVE/tree/lwjgl3/856-native-module-info-required-demo

The observed behavior confirms my expectation and is in line with descriptions of issues that have been reported in the past. Are you sure that you are running your application with a correctly configured modulepath? The only other way to solve this would be adding using the --add-modules command line flag when launching the JVM.

Hey @tlf30, @TheMrMilchmann,

This mechanism has failed in the past for unexpected user names (https://github.com/LWJGL/lwjgl3/issues/790) and interfering anti-virus systems (https://github.com/LWJGL/lwjgl3/issues/225) and I (personally) avoid relying on it. Instead, it is generally a good idea to set up your application in a way that the native libraries are loaded from a directory right away.

Custom packaging is always possible and recommended for developers that want full control. But it's quite a bit of extra work that needs to be done and thoroughly tested.

One easy solution to the issues mentioned is a custom -Dorg.lwjgl.system.SharedLibraryExtractPath. You can still rely on automatic natives extraction by LWJGL, but you force it to use a specific directory that you control and know won't cause issues. Could be as simple as your application's ./ and that should always be accessible (you'd have other kinds of trouble otherwise).

That's what I do in my LWJGL-based applications.

Any jar with natives contains a module info with a module name that is the same as other jars for natives of the same LWJGL module. When loading modules manually using ModuleLayer/ModuleFinder an exception (java.lang.module.FindException) is thrown due to duplicate module names.

This is by design. During development, there's no reason to download natives for other platforms (either directly or via Maven/Gradle). On end-user machines, the assumption is that there are platform/architecture-specific installers and they bundle or download specific natives. The only time you download all the natives is when building the installers (and even that is usually done on different machines or CI runners).

I have never had to add the native module names to my module-info.java. Do you have an example of this?

An example would be a jlink-packaged application. jlink requires all dependencies to be Java modules, you can't have non-modular jars on the classpath. In that case, you'd need to add the native modules in module-info. If the native module names had a platform/architecture suffix, you'd have to compile a different module-info class for each deployment target, which is not acceptable.

Note that native modules have a transitive dependency to the corresponding non-native module (and all non-core modules have a transitive dependency to the LWJGL core module). So, your application's module-info.java could have just:

requires org.lwjgl.jemalloc.natives;

instead of both:

requires org.lwjgl.jemalloc;
requires org.lwjgl.jemalloc.natives;

Going back to the original issue, another option you have is to merge the different native jars into one:

The internal directory structure allows this. Each binary is under a distinct directory for exactly this reason, to enable JAR merging.
The module-info.class under META-INF/versions/9 is exactly the same for all platforms/architectures, so you choose one and discard the rest.
The MANIFEST.MF file is also generic, except the LWJGL-Platform property. It's just informational though, LWJGL prints it only when something goes wrong with library loading. You can put something like multi-platform there or simply remove it, it's optional.
The INDEX.LIST file should be merged and contain entries for all files.

You could easily write a simple script that does the above automatically every time you download a new LWJGL distribution.

Thank you for the information @Spasi. I am currently traveling and will be home next week. I will post an example test case of the issue I am running into, as I am using LWJGLs native loader. The issue is when loading non-lwjgl jars that live in the same directly as lwjgl native jars.

I'll post an example next week once I get home.

OK, @TheMrMilchmann thank you for the example project. It has brought up many other issues. I started diving into the very old build configurations that the apps we are working on are built from. And the key difference is that our launchers are still using -jar <name> and the manifest files are still specifying a Class-Path for dependencies, because of this even though there is a module-info.java file present, much of the module system security is not enabled (but some parts still work at runtime, which is interesting)

While this opens a whole different can of worms that needs to be fixed. The underlaying issue is now: How can we package multiple platforms natives together in a single app and still use java modules? After I make the necessary corrections behind the scenes to get modules working at runtime (sigh) it will break the current loading process since we package all the natives together in our libs folders, and our plugins to our system package all the lwjgl natives together in their local lib folder.

Options I am currently looking at:

As @Spasi mentioned, repackaging natives into a single jar, which I would like to avoid as it would complicate an already complicated build system
Building an extractor that extracts the correct native jars from the plugin jars, which would be easy to add into the plugin system we have since we already extract other things based on the platform, but this does not fix the issue for the lwjgl natives we use in the core applications.

I will put some thought into this.

One initial thought I had, but have not tested yet.

Note that native modules have a transitive dependency to the corresponding non-native module (and all non-core modules have a transitive dependency to the LWJGL core module).

Would it be possible to reverse this? Perhaps give every native its own module name, then have those modules as a transitive static dependency to the java library jar. Example:

/*
 * Copyright LWJGL. All rights reserved.
 * License terms: https://www.lwjgl.org/license
 */
module org.lwjgl {
    requires transitive jdk.unsupported;

    //Natives
    requires transitive static org.lwjgl.natives.linux.arm32
    requires transitive static org.lwjgl.natives.linux.arm64
    requires transitive static org.lwjgl.natives.linux.x64
    requires transitive static org.lwjgl.natives.macos.arm64
    requires transitive static org.lwjgl.natives.macos.x64
    requires transitive static org.lwjgl.natives.windows.arm64
    requires transitive static org.lwjgl.natives.windows.x64
    requires transitive static org.lwjgl.natives.windows.x86

    exports org.lwjgl;
    exports org.lwjgl.system;
    exports org.lwjgl.system.jni;
    exports org.lwjgl.system.libc;
    exports org.lwjgl.system.libffi;
    exports org.lwjgl.system.linux;
    exports org.lwjgl.system.macosx;
    exports org.lwjgl.system.windows;
}

This would (in theory) eliminate the requirement for adding a requires for any of the natives. I did a quick compile test and a static transitive does compile but I have not tested runtime behavior yet.

Hey @tlf30,

I believe the suggestion above would still require you to add the natives dependency in some other way (e.g. with --add-modules). A requires static dependency is the same as just requires at compile time, but it is mostly ignored at runtime. The module system will not add any of those dependencies in the module graph automatically.

I don't like the current situation either, but it felt like the least complicated/verbose in practice. I don't think the module system is designed to handle this particular case. But, honestly, it's not a big deal, we made it work.

Also, I don't see how changing the module descriptors would help you with the original issue of bundling multiple natives. I'm a fan of lazy solutions, just "throw them all in there", but you can't be lazy in this case, you'll have to do some work to make it work. But if you're going to do extra work, why not do it properly and only ship the natives required for the target installation only?

LWJGL / lwjgl3