Implement caching

[bmarwell]

Rationale

On short-lived processes, do not have the overhead of extracting files every single time the library is used for the first time.

The original request is a bit harsh (and even offensive), but the actual message seems to be correct to me: https://github.com/java-native/libloader/issues/4#issuecomment-581005289

Design / Implementation

https://github.com/java-native/libloader/wiki/Extraction-Process

[bmarwell]

Hi @tresf,

the "legacy" code from sc3amer you mentioned does all of this by its own.

This project tries to mimic the current situation. This means it already works.

The reason I introduced the scijava/native-lib-loader was to separate dll loading logic from the actual serial library.

libLoader.loadLibrary("jssc", SerialNativeInterface.getNativeLibraryVersion()); The cart is before the horse in the above instance, getNativeLibraryVersion is a JNI call, it's not available yet!

Yes. that wouldn't work and tbh: I wouldn't use it. In fact, for jssc, I would just leave out the version number from the file name as in the current branch.

Instead, hard-coding it adds an additional maintenance point as the version needs to be constantly updated in the native project with leverages it.

Yes and no, depends on what you are trying to achieve. I can be an advantage. If you update the library version manually, a user couldn't accidently mix a wrong version of a native jar with the actual library.

But the new artifact jssc-all I was going to create would actually pull in jssc-native-<os>-<arch> (all native dependencies) with the same project version. Thus, when using a dependency manager, you would never be able to accidently mix up versions.

To test a patched version of that same library (since the patch must now completely recompile the native project which leverages it) will often require recompilation (or at least re-building) of the dependant app, or clobber the cached library with a patched version (which could lead to versioning confusion).

You can just do mvn install for your own OS in the project java-native/jssc-native. The next time you start your unit test, your library would be used. I haven't written a how-to yet, but I designed it in a way that this should become way easier. Promised.

---

For jssc, a typical goal

• just leave the version out or
• read the version from a maven-generated .properties file

The current implementation (extracting to /tmp) doesn't need versioning because there could be no clash of file names due to unique folder names.

---

There is also another way.

1. we talked about caching.
2. we talked about using /tmp.
3. we haven't talked about using -Djava.library.path yet.

So, if you are concerned about both name clashing and IO when using /tmp, a very natural way I was going to implement would be using the JVM startup option -Djava.library.path. In the application you ship you could have the natives extracted to a directory, let's say $APPDIR/libjssc/linux-x86_64-64/libjssc.so. When your startup script (on linux maybe startapp.sh) would detect linux amd64, it would execute this option: java -Djava.library.path=$APPDIR/libjssc/linux-x86_64-64/libjssc.so -jar yourapp.jar.

Actually, the whole idea of this project (and of the original library from scijava) was to be able to not need to fiddle with this and having the dlls loaded from a jar file, even if this means extracting them first.

I do not know your production application and I cannot say which way would be the best for you, though. All I can do is to write down the three possibilities I know and their advantages and disadvantages to help you choose.

If you need more information, don't hesitate to ask.

[tresf]

You can just do mvn install for your own OS in the project java-native/jssc-native. The next time you start your unit test, your library would be used. I haven't written a how-to yet, but I designed it in a way that this should become way easier. Promised.

I think it's a bit short-sighted to think that all dependent projects intend to rebuild everything to replace a native library. Often, native libraries are patched in for proof-of-concept, but I've already explained this point. Your answer simply explains the existing limitations that are already present, which do not address the concern.

For example, if you use an application that has 5 separate native libraries... and as and end-user are trying to patch a small bug with just one of these libraries... (a very common problem) your recommendation is for that person to 1. Rebuild the native library using Maven. 2. Update the artifact (manually) in the dependant library. 3. Rebuild the dependant library using whatever system.

What inspired jssc's fork was to be able to get it working with a more modern Windows JDK runtime (the upstream build would segfault), and that fork successfully did just that, but was done by using a C compiler and swapping the library out of place until the crash went away.

Then, once the crash went away, bring backwards compatibility with older Windows OSs needed to do the same technique.

I don't think it's responsible to minimize the usefulness of persistent library storage unless you have real-world experience patching obscure JNI build bugs.

[bmarwell]

I'm still afraid that this technique will pass the burden to someone without knowledge or control over system properties. I'm also afraid that it may be misusing the JDK feature. If any project uses this value internally, you may be causing unintended problems (granted downstream should check using that property, but it's hard to control what downstream does).

Hi @tresf,

as I commented in your issue at tray, setting t he java.library.path has no effect anyway. If any tool uses this value internally, it never should have worked (it probably didn't work anyway). And as there can only be one external start script which knows all the library locations, there is no possibility to cause unintended problem.

Your assumption would be right if setting this var worked in the JVM. But since it does not (as per specification), I am not going to bother with this unspecified behaviour.

---

I think it's a bit short-sighted to think that all dependent projects intend to rebuild everything to replace a native library.

They don't need to rebuild "everything". Just a single native library on a single OS.

E.g. they can clone the new jssc-native repository, modify the C code, run mvn install and it will work locally. They can send us either the jar file which got installed to $HOME/.m2/repository or they can send us the patch for testing.

Often, native libraries are patched in for proof-of-concept, but I've already explained this point. Your answer simply explains the existing limitations that are already present, which do not address the concern.

No, because your previous assumption is wrong. See previous paragraph.

For example, if you use an application that has 5 separate native libraries... and as and end-> user are trying to patch a small bug with just one of these libraries... (a very common problem) your recommendation is for that person to

1. Rebuild the native library using Maven.

… rebuild and install to their local repository, that is.

2. Update the artifact (manually) in the dependant library.

No. It is not inside the other library, it is just another file NEXT to the dependent library. This is the same mis-assumption you made earlier. You just need to build the jssc-native-os-arch.jar. No need to rebuild everything else.

3. Rebuild the dependant library using whatever system.

No, same as 2.

This makes it even easier then the current build system, as you do not need to build all the java code as well and wait for all the tests to pass.

Even better, you could just use CMAKE to build the new native library, add it to java.library.path and remove the faulty native-jssc-linux-x86_64-64.jar. No need to maven-install.

I created way more possibilites than we have at the moment with this library, but I think you just do not see them yet. That is no offense intended -- I am just pretty sure I thought this through.

Do you need a sample application? I can set up one for you to see some example workflows.

---

Example workflow

You are shipping program X, which in turn uses jssc.

example programme

You will have this folder structure:

$APP/
  - app.jar
  - start.sh
  - lib/
     - jssc-header.jar
     - jssc-lib.jar (java-code)
     - jssc-native-linux-x86_64-64.jar
     - jssc-native-otheros-otherarch-bitness.jar

possibility 1

To try your fix, just checkout the jssc-native repository, run mvn package after modifying the C files and copy the jar which was created to $APP/lib/, overwriting the existing jar.

possibility 2

If you do not want to invoke maven (which will merely invoke the C compiler in the new project), you can just remove the existing jssc-native-linux-x86_64-64.jar and modify the start.sh script to receive -Djava.libary.path=path/to/new/libjssc.so.

summary

I think these methods are quite conventient and are in fact only one-step setups.

[bmarwell]

There will be also a single jar dependency if you like. Had this in mind anyway, so one dependency will not pull in multiple others.

[tresf]

I've remove my conversation, but kept it for archival purposes. I'm unsubscribing from this topic. libloader_#6.txt