search

herumi / mcl

a portable and fast pairing-based cryptography library
BSD 3-Clause "New" or "Revised" License
450 stars 152 forks source link

Java FFI Makefile does not work correctly on Mac OS #105

Closed rheitjoh closed 3 years ago

rheitjoh commented 3 years ago

Problem

The Java FFI Makefile (under /ffi/java/Makefile) is missing an include path for jni_md.h for Mac OS. This file–on Mac OS–is stored under $(JAVA_INC)/darwin which is not added as a CFLAG. The compilation therefore fails due to not being able to find jni_md.h.

How to fix

Depending on the OS, either $(JAVA_INC)/linux or $(JAVA_INC)/darwin should be added to the CFLAGS.

Example fix moving the addage of the include path into the OS condition:

ifeq ($(UNAME_S),Darwin)
  JAVA_INC?=-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/JavaVM.framework/Headers/
  CFLAGS+=$(JAVA_INC)/darwin
else
  JAVA_INC?=-I/usr/lib/jvm/default-java/include
#JAVA_INC=-I/usr/lib/jvm/java-7-openjdk-amd64/include
  CFLAGS+=-z noexecstack $(JAVA_INC)/linux
  LDFLAGS+=-lrt
endif
CFLAGS+=$(JAVA_INC) -I $(TOP_DIR)/include -I $(TOP_DIR)/../xbyak -I $(TOP_DIR)/../cybozulib/include -Wno-strict-aliasing

I can create a pull request for this, but due to #104 the Java FFI is not working atm.

herumi commented 3 years ago

I tested make test at ffi/java on Intel macOS and it runs well. What error is there?

% uname -a
Darwin macpro.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64

mcl/ffi/java% make test_mcl
mkdir -p com/herumi/mcl
...
OK : verify signature
all test passed

a.log

rheitjoh commented 3 years ago

I personally don't have a Mac, but the error was that jni_md.h cannot be found which makes sense cause that file is contained in $(JAVA_INC)/darwin (at least on my colleague's Macs). That path is missing in the Makefile.

Where is your jni_md.h and which Java version are you using? I wonder if they changed the location of jni_md.h for some Java version.

feidens commented 3 years ago

Hi from the Mac colleague,

I think the assumption that I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/JavaVM.framework/Headers/ exists is false. Usually it does not exists if you just install a jdk. (This looks more related to command line tools that Xcode installs, but also xcode-select --installdoes not help, so 🤷 )

JanBobolz commented 3 years ago

Hi, other mac colleague here (Big Sur Intel, openJDK 14)

Regarding the error in the ticket

We tend to use the path returned by /usr/libexec/java_home, which is /Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home in my case, for JAVA_INC.

With the fix provided by @rheitjoh above, make JAVA_INC=-I/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include in mcl/ffi/java works. Without it, the command fails with the following error

jan@Jans-MBP java % make JAVA_INC=-I/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include
mkdir -p com/herumi/mcl
swig -java -package com.herumi.mcl -outdir com/herumi/mcl -c++ -Wall elgamal.i
c++ elgamal_wrap.cxx -o ../../lib/libmclelgamaljava.dylib -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/openssl/include -I/usr/local/opt/gmp/include -g3 -Wall -Wextra -Wformat=2 -Wcast-qual -Wcast-align -Wwrite-strings -Wfloat-equal -Wpointer-arith -Wundef -m64 -I include -I test -fomit-frame-pointer -DNDEBUG -fno-stack-protector -O3  -DMCL_USE_VINT -DMCL_DONT_USE_OPENSSL -fPIC -I/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include/linux -I ../../include -I ../../../xbyak -I ../../../cybozulib/include -Wno-strict-aliasing ../../lib/libmcl.a -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/gmp/lib   -m64  -shared
In file included from elgamal_wrap.cxx:178:
/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include/jni.h:45:10: fatal error: 
      'jni_md.h' file not found
#include "jni_md.h"
         ^~~~~~~~~~
1 error generated.
make: *** [../../lib/libmclelgamaljava.dylib] Error 1

This is because the directory layout of /Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include is

jan@Jans-MBP include % ls -R
classfile_constants.h   jdwpTransport.h     jvmticmlr.h
darwin          jni.h
jawt.h          jvmti.h

./darwin:
jawt_md.h   jni_md.h

i.e. the jni_md.h file is under $(JAVA_INC)/darwin (whereas your current Makefile only includes $(JAVA_INC)/linux)

Regarding your default JAVA_INC:

On my machine, the following file exists: /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni.h

However, /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk (which is a prefix of the path you use as your default JAVA_INC) doesn't point to /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk but to /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk (i.e. a newer version), which (at least for me) does not have the JavaVM.framework directory and does not contain any jni.h anywhere (I can offer a /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk/System/Library/Frameworks/JavaNativeFoundation.framework/Versions/A/Headers/JNFJNI.h, but ... not sure what I'm supposed to think about that).

If I use the MacOSX10.15.sdk path explicitly for JAVA_INC, compilation works with warnings, but tests succeed.

jan@Jans-MBP java % make test_mcl JAVA_INC=-I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers
mkdir -p com/herumi/mcl
swig -java -package com.herumi.mcl -outdir com/herumi/mcl -c++ -Wall mcl.i
c++ mcl_wrap.cxx -o ../../lib/libmcljava.dylib -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/openssl/include -I/usr/local/opt/gmp/include -g3 -Wall -Wextra -Wformat=2 -Wcast-qual -Wcast-align -Wwrite-strings -Wfloat-equal -Wpointer-arith -Wundef -m64 -I include -I test -fomit-frame-pointer -DNDEBUG -fno-stack-protector -O3  -DMCL_USE_VINT -DMCL_DONT_USE_OPENSSL -fPIC -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/linux -I ../../include -I ../../../xbyak -I ../../../cybozulib/include -Wno-strict-aliasing ../../lib/libmcl.a -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/gmp/lib   -m64  -shared
In file included from mcl_wrap.cxx:178:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni.h:203:6: warning: 
      'TARGET_RT_MAC_CFM' is not defined, evaluates to 0 [-Wundef]
#if !TARGET_RT_MAC_CFM && defined(__ppc__)
     ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni.h:756:6: warning: 
      'TARGET_RT_MAC_CFM' is not defined, evaluates to 0 [-Wundef]
        #if TARGET_RT_MAC_CFM && defined(__ppc__)
            ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni.h:1886:6: warning: 
      'TARGET_RT_MAC_CFM' is not defined, evaluates to 0 [-Wundef]
#if !TARGET_RT_MAC_CFM && defined(__ppc__)
     ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni.h:1900:5: warning: 
      'TARGET_RT_MAC_CFM' is not defined, evaluates to 0 [-Wundef]
#if TARGET_RT_MAC_CFM && defined(__ppc__)
    ^
4 warnings generated.
javac MclTest.java
cd ../../lib && LD_LIBRARY_PATH=../lib: java -classpath ../ffi/java MclTest
libName : libmcljava.dylib
curve=BN254
OK : x != y
OK : x == z
OK : x == 5
[...]

Summary

So overall, it would be cool if you could apply the fix provided by @rheitjoh so that we have the option to use the openJDK jni files.

herumi commented 3 years ago

Thank you for the report. I can understand it. I add JAVA_INC_DIR to Makefile. It runs well for both openjdk 12 and 15. Could you try https://github.com/herumi/mcl/commit/a06bfd3b8f39b967071e8758ca76ae31268ab81c#diff-dbd98d720cb11066d7e6b68a132cec80651f1099c09eefc3a03263f7cfc0bd39 ?

make test_mcl JAVA_INC_DIR=/Library/Java/JavaVirtualMachines/openjdk-12.0.1.jdk/Contents/Home/include/
make test_mcl JAVA_INC_DIR=/usr/local/opt/openjdk/include/
ls /usr/local/opt/openjdk@15/include/
./                      classfile_constants.h@  jawt_md.h@              jni.h@                  jvmti.h@
../                     jawt.h@                 jdwpTransport.h@        jni_md.h@               jvmticmlr.h@
ls /Library/Java/JavaVirtualMachines/openjdk-12.0.1.jdk/Contents/Home/include/
./                      classfile_constants.h   jawt.h                  jni.h                   jvmticmlr.h
../                     darwin/                 jdwpTransport.h         jvmti.h
JanBobolz commented 3 years ago 
jan@Jans-MBP java % make test JAVA_INC=-I/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include
[...]
all test passed

Works for me now. Thanks! edit: just noticed the JAVA_INC_DIR. Both make test_mcl JAVA_INC_DIR=/usr/local/opt/openjdk/include/ and make test_mcl JAVA_INC_DIR=/Library/Java/JavaVirtualMachines/openjdk-14.0.1.jdk/Contents/Home/include work for me, too.