Space in directory name and read only during build

[KyronSr]

Trying to use this, I hit a problem where if your jar is located in a path with a space in the name (on Windows at least), it would throw an error: java.net.URISyntaxException: Illegal character in opaque part at index This is because of Loader.java, around here (line 1551 in latest):

for (URL url : urls) { URI uri = url.toURI();

Locally I worked around this by changing it to:

for (URL url : urls) { String tUrl = url.toString().replace(" ","%20"); URI uri = new URI(tUrl);

I'm sure there are more elegant solutions but this got me going again.

Also, during the build process, I had some DLLs which were read only. This caused the build to crash when it was copying the various DLLs (line 1219). Changed it to simply make the file writeable before copying in:

// make sure it isn't read only if (fo.exists()) { fo.setWritable(true); }

One last thing -- a few line blurb on the front-page on how to build it would be nice (I'm not a maven god by any means but I was able to figure it out so it isn't hard but one sentence with an example command line would be nice)

[saudet]

Trying to use this, I hit a problem where if your jar is located in a path with a space in the name (on Windows at least), it would throw an error: java.net.URISyntaxException: Illegal character in opaque part at index This is because of Loader.java, around here (line 1551 in latest):

for (URL url : urls) { URI uri = url.toURI();

Locally I worked around this by changing it to:

for (URL url : urls) { String tUrl = url.toString().replace(" ","%20"); URI uri = new URI(tUrl);

I'm sure there are more elegant solutions but this got me going again.

I'm not able to reproduce this. It works just fine for me with JAR files in directories with spaces in them, either on Linux or Windows. Please provide some way to reproduce this.

Also, during the build process, I had some DLLs which were read only. This caused the build to crash when it was copying the various DLLs (line 1219). Changed it to simply make the file writeable before copying in:

// make sure it isn't read only if (fo.exists()) { fo.setWritable(true); }

That's more of a feature than a bug. I don't think users would be happy to have their read-only files overwritten by their programs!

One last thing -- a few line blurb on the front-page on how to build it would be nice (I'm not a maven god by any means but I was able to figure it out so it isn't hard but one sentence with an example command line would be nice)

I think this is what you are looking for: https://github.com/bytedeco/javacpp-presets/wiki/Create-New-Presets There's a link in the "Getting Started" section of the README.md file. There's enough stuff in there already, and people that are smart enough to modify the code can usually figure out how to build it as well, even it's just a click away!

[KyronSr]

On the "read only" -- this is part of the build process for javacpp (not preset) where it copies DLLs into the target folder. If those source files are read only, the second build may fail. Maybe the code should check to see if the destination is read-only before trying to overwrite it (which results in the build crashing with an AccessDenied error). (I was simply running mvn package)

On the spaces in the URL bit -- this piece of code shows what I am talking about:

// throws error java.net.URISyntaxException: Illegal character in path at index 16 String url = "file:/C:/Program Files/Test/lib/tensorflow-core-api-0.1.0-SNAPSHOT-windows-x86_64.jar"; // this URL does not (no space) //String url = "file:/C:/temp/lib/tensorflow-core-api-0.1.0-SNAPSHOT-windows-x86_64.jar"; URL aUrl = new URL(url); URL[] urls = new URL[1]; urls[0] = aUrl; org.bytedeco.javacpp.Loader.loadLibrary(org.tensorflow.internal.c_api.global.tensorflow.class, urls, "");

(Note, the class won't load but it shows the URL error).

[saudet]

On the "read only" -- this is part of the build process for javacpp (not preset) where it copies DLLs into the target folder. If those source files are read only, the second build may fail. Maybe the code should check to see if the destination is read-only before trying to overwrite it (which results in the build crashing with an AccessDenied error). (I was simply running mvn package)

Again, which DLLs that get copied from where? I've never encountered such a scenario. I can't fix what isn't broken! You'll need to give me something to work with.

[saudet]

On the spaces in the URL bit -- this piece of code shows what I am talking about:

Please give me a piece of code that fails. I cannot fix what isn't broken.

[KyronSr]

On the spaces in the URL bit -- this piece of code shows what I am talking about:

Please give me a piece of code that fails. I cannot fix what isn't broken.

I pasted some code in my reply but it got formatted badly. I'll give it another try here (I just included this in a hello world java file):

// throws error java.net.URISyntaxException: Illegal character in path at index 16 
String url = "file:/C:/Program Files/Test/lib/tensorflow-core-api-0.1.0-SNAPSHOT-windows-x86_64.jar";
// this URL does not (no space) 
//String url = "file:/C:/temp/lib/tensorflow-core-api-0.1.0-SNAPSHOT-windows-x86_64.jar";
URL aUrl = new URL(url);
URL[] urls = new URL[1];
urls[0] = aUrl;
org.bytedeco.javacpp.Loader.loadLibrary(org.tensorflow.internal.c_api.global.tensorflow.class, urls, "");

The first URL throws an immediate exception because .toURI doesn't like spaces.

As for the "read only" aspect of this -- the Builder.java, where is does

if (copyLibs)

It takes a copy of various libraries which it needs (for instance, I'm compiling this on Windows for x86,x64 and it copies them into the target\classes\org\bytedeco\javacpp\windows-x86_64 folder). An example would be vcruntime140.dll. It appears to have copied it from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT (which seems a sensible place since it is a redist) but the source file is read only. Since it is read only, it copies it into the "target..." folder with the read-only attrib set. The next time it tries to copy it, it crashes because it doesn't check if it needs to and\or if the destination is readonly (around line 1218):

                       File fo = new File(directory, fi.getName());
                        if (fi.exists() && !outputFiles.contains(fo)) {

[saudet]

Ah, you're passing a custom URL. It will need to be a valid URL, yes, it can't be broken, that's normal.

[saudet]

It takes a copy of various libraries which it needs (for instance, I'm compiling this on Windows for x86,x64 and it copies them into the target\classes\org\bytedeco\javacpp\windows-x86_64 folder). An example would be vcruntime140.dll. It appears to have copied it from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT (which seems a sensible place since it is a redist) but the source file is read only. Since it is read only, it copies it into the "target..." folder with the read-only attrib set. The next time it tries to copy it, it crashes because it doesn't check if it needs to and\or if the destination is readonly (around line 1218):

The copy operation does not copy the attributes, so those files cannot be read-only. If they end up being read-only though, this is a bug in the implementation of Java SE that you are using, and you should report that bug upstream.

[saudet]

You're right. I've just tested on Windows with Oracle JDK 8 and Files.copy() incorrectly copies the attributes of the files. If you could report this bug upstream, that would be great! Thanks

[KyronSr]

You're right. I've just tested on Windows with Oracle JDK 8 and Files.copy() incorrectly copies the attributes of the files. If you could report this bug upstream, that would be great! Thanks

Both OpenJDK and Oracle JDK (and I suspect others) call out to the windows API to copy the files. As part of this, file attributes are copied. If you look at the WindowsFileCopy.java, it simply calls out to WinAPI's CopyFileEx and CopyFileEx copies these types of attributes. I doubt they would classify this as a bug as it has always worked this way (and I doubt they would be inclined to "fix" it as the ramifications for making a change like this would be pretty drastic).

[KyronSr]

Ah, you're passing a custom URL. It will need to be a valid URL, yes, it can't be broken, that's normal.

Actually what I was doing above was to give you an example of how it was failing. I'm actually using the Java TensorFlow library which uses your library to call into the C++ TensorFlow DLL. A further complication is that I'm wrapping all of this in a Pentaho Kettle plugin. I'm not constructing the URL (nor is the TenorFlow library doing so).

The actual stack trace of the problem I've hit is:

java.lang.UnsatisfiedLinkError: java.net.URISyntaxException: Illegal character in opaque part at index 20: jar:file:/C:/Program Files/Kettle/data-integration/plugins/steps/AlterianTensorFlow/lib/tensorflow-core-api-0.1.0-SNAPSHOT-windows-x86_64.jar!/org/tensorflow/internal/c_api/windows-x86_64/jnitensorflow.dll at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1652) at org.bytedeco.javacpp.Loader.load(Loader.java:1265) at org.bytedeco.javacpp.Loader.load(Loader.java:1109) at org.tensorflow.internal.c_api.global.tensorflow.(tensorflow.java:12) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.bytedeco.javacpp.Loader.load(Loader.java:1176) at org.bytedeco.javacpp.Loader.load(Loader.java:1125) at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48) at org.tensorflow.TensorFlow.init(TensorFlow.java:111)

I've dug into the TensorFlow library's code and they seem to only be doing:

''' org.bytedeco.javacpp.Loader.load(org.tensorflow.internal.c_api.global.tensorflow.class); '''

Somwhere it is crafting a URL which has a space in it (which doesn't like being converted to a URI). Looking at the tensorflow.java (which is a preset), there are paths there in the preload path, etc. I wonder if maybe that might be the source of the issue. I'll take a look at the preset and see if I fix the paths in there if it solves the problem or not.

[saudet]

Would you be able to share some code snippet using TensorFlow that fails?

[saudet]

FWIW, I suspect there is a bug in Pentaho Kettle that shows up only with JavaCPP for some reason. There is probably a way to create a workaround for this, but I won't be able to unless you provide me with a way to reproduce this issue!