search

SWI-Prolog / packages-jpl

JPL: The Prolog <-> Java interface
BSD 2-Clause "Simplified" License
54 stars 33 forks source link

The specified procedure could not be found #82

Closed Venorcis closed 4 years ago

Venorcis commented 4 years ago

With SWI Prolog 8 (either 8.2.1-1 or 8.3.7-1) on 64-bit Windows 10, I am getting the following error when trying to System.load the jpl.dll: "java.lang.UnsatisfiedLinkError: jpl.dll: The specified procedure could not be found."

This is using Java 1.8.0_251-b08 (64-bit). The problem does not occur when doing exactly the same with SWI Prolog 7.6.4.

I analysed the file using https://github.com/lucasg/Dependencies, and that indicates no errors at all, so to me that seems to point to some incompatibility/misuse of a JVM function.

JanWielemaker commented 4 years ago

I guess we cannot get any idea which "specified procedure" we are talking about? Did you make sure you copied all the other support dlls from the bin directory of that Prolog version? Notably that you are not using old MinGW support libraries?

You may also want to try the examples in doc/packages/examples/jpl/java in the Prolog installation.

Venorcis commented 4 years ago

I guess we cannot get any idea which "specified procedure" we are talking about?

Sadly, no. I tried to get this information, but these UnsatisfiedLinkErrors are terrible...

Did you make sure you copied all the other support dlls from the bin directory of that Prolog version? Notably that you are not using old MinGW support libraries?

Well we only use the direct dependencies, which are indeed copied from the SWI installation. We do this without issue for SWI 7. These are essentially the Java calls done now which always gives me the same error on the final line (loading jpl.dll):

System.load(dir + "libwinpthread-1.dll");
System.load(dir + "libgcc_s_seh-1.dll");
System.load(dir + "libgmp-10.dll");
System.load(dir + "zlib1.dll");
System.load(dir + "libswipl.dll");
System.load(dir + "jpl.dll");

Again it's very weird that the DLL Dependency Analyzer gives the green light on everything, but then somehow some procedure is missing eventually; it's quite a specific error that does seem to indicate everything is loaded as it should be, but something is incompatible, which I suspect is then most likely in relation to the JVM, especially because libswipl itself loads just fine.

JanWielemaker commented 4 years ago

Sadly, no. I tried to get this information, but these UnsatisfiedLinkErrors are terrible...

I know. On Linux I would try tracing dlsym() using gdb. I don't know whether you can do something similar on Windows. Probably yes ...

I suspect is then most likely in relation to the JVM, especially because libswipl itself loads just fine.

One possible explanation is that jpl.dll is compiled against the OpenJDK rather than the Oracle/SUN JDK. I don't know how to verify this. This was necessary as recent Oracle JDKs do not install in Wine (used for the cross compilation build) and Oracle JDK cannot legally be included.

Maybe ask on our forum?

JanWielemaker commented 4 years ago

See also https://github.com/SWI-Prolog/packages-jpl/issues/34

I've tried using Oracle Java 8 and 13. In both cases all runs fine when I load the jvm into Prolog (?- [library(jpl)].), but trying to run the tests that embed Prolog into Java fail. On both version is doesn't get anywhere and crashes almost immediately on an access violation (5). According to the crash dumps it did finish loading the Prolog DLLs.

How do we get more info?

Venorcis commented 4 years ago

See also #34

I've tried using Oracle Java 8 and 13. In both cases all runs fine when I load the jvm into Prolog (?- [library(jpl)].), but trying to run the tests that embed Prolog into Java fail. On both version is doesn't get anywhere and crashes almost immediately on an access violation (5). According to the crash dumps it did finish loading the Prolog DLLs.

How do we get more info?

Which Java version were you using precisely? Anything newer than 8u251 is indeed known to us to give EXCEPTION_ACCESS_VIOLATIONs (with JPL but also with other JNI libraries, see e.g. https://hi.service-now.com/kb_view.do?sysparm_article=KB0853593). It's very likely related to their move to Visual Studio 2017 ("Native applications that have depended on and assumed the presence of MSCVR100.dll in the JDK/JRE directory will fail to run"), see https://www.oracle.com/java/technologies/javase/8u261-relnotes.html

JanWielemaker commented 4 years ago

Which Java version were you using precisely

jdk-8u261-windows-x64.exe. I think that was the only option available for JDK 8? The link indeed indicates one needs 8u251 :cry:

We do not have dependencies on MSCVR as we use MinGW for compiling with comes with its own C runtime. Considering that we can embed the JVM into SWI-Prolog, I'd assume things basically work together.

Venorcis commented 4 years ago

Which Java version were you using precisely

jdk-8u261-windows-x64.exe. I think that was the only option available for JDK 8? The link indeed indicates one needs 8u251 😢

We do not have dependencies on MSCVR as we use MinGW for compiling with comes with its own C runtime. Considering that we can embed the JVM into SWI-Prolog, I'd assume things basically work together.

Indeed not on MSCVR100.dll exactly, but JPL.dll does seem to depend on MSVCRT.dll, which might be related? I'm not into this C/DLL stuff at all ;) image

P.S. If needed, older Java 8 versions can be downloaded at https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html

JanWielemaker commented 4 years ago

but JPL.dll does seem to depend on MSVCRT.dll, which might be related

Indeed, my mistake. MinGW links to MSVCRT.dll. I see some mentions linking to MSVCRT100.dll, but that seems legally complicated :cry:

Thanks for the download link. I already got three Java versions more than this morning :smile: OpenJDK 14 shows the same crash as the Oracle ones, which makes it likely we are in some C runtime library hell :cry:

JanWielemaker commented 4 years ago

The good news is that the demo programs embedding version 8u251 now work. My new one-to-one thread mapping seems to work mostly as well, though it prints a message it shouldn't. I'll try to find out why.

Does your code involve other JNI components? Could we be looking at something that happens after Prolog was loaded? Loading a dll should not result in undefined procedures in the first place AFAIK.

Venorcis commented 4 years ago

The good news is that the demo programs embedding version 8u251 now work. My new one-to-one thread mapping seems to work mostly as well, though it prints a message it shouldn't. I'll try to find out why.

Does your code involve other JNI components? Could we be looking at something that happens after Prolog was loaded? Loading a dll should not result in undefined procedures in the first place AFAIK.

The example code I posted above already causes the error for me (so just calling System.load on jpl.dll) with SWI 8. I attached it fully here: JPLtest.zip

JanWielemaker commented 4 years ago

This is getting rather mysterious. Yip, I can reproduce this. If I however copy jpl.dll from my build directory all loads just fine!? The one from the 8.3.7 release is indeed broken, but it is compiled in the same docker image!?

Anyway, you should be able to download the current version tomorrow from the download page at https://www.swi-prolog.org/download/daily/bin/ Verify it is really Sep 11. If not, the nightly build failed.

Venorcis commented 4 years ago

This is getting rather mysterious. Yip, I can reproduce this. If I however copy jpl.dll from my build directory all loads just fine!? The one from the 8.3.7 release is indeed broken, but it is compiled in the same docker image!?

Anyway, you should be able to download the current version tomorrow from the download page at https://www.swi-prolog.org/download/daily/bin/ Verify it is really Sep 11. If not, the nightly build failed.

Morning Jan :)

I tried the DLLs from the swipl-w64-2020-09-11.exe installer. Same "procedure could not be found" error I'm afraid :(

JanWielemaker commented 4 years ago

Interesting. Didn't try yet. I have a telco in a couple of minutes. Will look afterwards. One starts thinking something weird happens during the packaging!?

Venorcis commented 4 years ago

Interesting. Didn't try yet. I have a telco in a couple of minutes. Will look afterwards. One starts thinking something weird happens during the packaging!?

Sorry, I spoke too soon! I have to check our code that updates our local SWI install, because it didn't properly update (I did clear this multiple times in the past few days, but forgot to do it this morning). It does actually work now with the latest build, although not all our tests succeed (but that's probably because of the SWI 8 changes).

Venorcis commented 4 years ago

Is it correct that the win32 build does not contain a jpl.dll?

And I tried just replacing our dynamic declarations with thread_local internally/automatically, but that gives unexpected results (i.e. failing end-tests) upon runtime I'm afraid. I'll have to dig into that more later...

JanWielemaker commented 4 years ago

Is it correct that the win32 build does not contain a jpl.dll?

Yes. The reason for that is that there is no 32 bit Java that can be installed in the cross-compiling environment AFAIK. Currently OpenJDK is used to build jpl. OpenJDK is easily automatically downloaded and installed in the docker, but has no 32-bit version. But then, I think there are very few 32-bit Windows machines around anymore. At least, nobody protested after the announcement to drop the 32-bit jpl.

And I tried just replacing our dynamic declarations with thread_local

This is going off topic. I'll answer on mail.

I think we can close this. Resume: use Java 8u251and somehow the jpl.dll of the latest release is broken. If that happens again we have some clue where to start looking :smile:

Venorcis commented 4 years ago

@JanWielemaker I was actually discussing this with Koen, we kind of 'need' 32-bit support to be able to interface with StarCraft (which uses some JNI-API of its own). This is also one of the reasons we have not (yet) upgraded beyond Java 8. However, in a recent search, I did figure out that recently the AdoptOpenJDK initiative is actually building 32-bits Windows versions of the OpenJDK, see https://adoptopenjdk.net/releases.html

JanWielemaker commented 4 years ago

That was easy. Extended the Dockerfile a little and it now produces a 32-bit jpl.dll :smile: Should be in tomorrow's nightly build. Send me a mail if you want an early copy.

Venorcis commented 4 years ago

That was easy. Extended the Dockerfile a little and it now produces a 32-bit jpl.dll 😄 Should be in tomorrow's nightly build. Send me a mail if you want an early copy.

Great! No hurry; I'll await the build. I probably only have time to test this coming Tuesday (with StarCraft then perhaps even).

Should I make another issue for the Java support beyond 8u251? I assume the same access violation errors occur with the latest Oracle Java 11 for example? If possible we'd really like to move to Java 11 (the latest LTS version) with GOAL now.

JanWielemaker commented 4 years ago

Should I make another issue for the Java support beyond 8u251?

Is already created as https://github.com/SWI-Prolog/packages-jpl/issues/34. Considering this seems to happen pretty much the same on OracleJDK and OpenJDK and what we discussed, I'm starting to fear this could be a MinGW vs. recent MSVC issue (likely related to the C runtime library).

Hopefully there is or will be a solution for that. Switching to MSVC is barely an option. It is quite a bit of work to get all dependencies right and the result is about 50% slower.