Closed mstoodle closed 5 years ago
A first-cut porting outline to kick-start some conversation:
I started playing with this two nights ago. Please assign to me and I'll have 2 wip pull requests and while I can certainly take comments, there are changes that I have to go back and investigate/resolve still. These changes do not compile as yet. I believe I get through a lot (if not all) of java compilation and but I am failing in compiling openj9 native code. I get through OMR native compile through so I believe I have the glue layer correct.
My current failure is looking like this :
/Library/Developer/CommandLineTools/usr/bin/make -C ./jilgen -f constgen.mk
cc -DJAVA_SPEC_VERSION=9 -O3 -fno-strict-aliasing -g -MMD -DOSX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -fPIC -DIPv6_FUNCTION_SUPPORT -DJ9HAMMER -m64 -Wimplicit -Wreturn-type -Werror -I. -I../include -I../oti -I../nls -I/Users/joe_dekoning/openjdk/build/macosx-x86_64-normal-server-release/vm/omr/include_core -DUT_DIRECT_TRACE_REGISTRATION -DTR_HOST_X86 -c -o copyright.o ../makelib/copyright.c
cc -DJAVA_SPEC_VERSION=9 -O3 -fno-strict-aliasing -g -MMD -DOSX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -fPIC -DIPv6_FUNCTION_SUPPORT -DJ9HAMMER -m64 -Wimplicit -Wreturn-type -Werror -I. -I../include -I../oti -I../nls -I/Users/joe_dekoning/openjdk/build/macosx-x86_64-normal-server-release/vm/omr/include_core -DUT_DIRECT_TRACE_REGISTRATION -DTR_HOST_X86 -c -o jilconsts.o jilconsts.c
cc -L. -L../ -L../lib/ \
copyright.o jilconsts.o \
-Xlinker --start-group \
-lj9exelib -lj9utilcore -lj9prtstatic -lj9hashtable -lj9avl -lj9pool -lomrglue \
-Xlinker --end-group \
-lj9thr29 \
-o ../constgen -lm -lpthread -lc -lrt -ldl -lutil -Wl,-z,origin,-rpath,\$ORIGIN,--disable-new-dtags,-rpath-link,../
ld: unknown option: --start-group
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[5]: *** [../constgen] Error 1
make[4]: *** [constgen] Error 2
make[3]: *** [build-j9] Error 2
make[2]: *** [j9vm-build] Error 2
Also need changes from https://github.com/ibmruntimes/openj9-openjdk-jdk9/pull/59
Follow on #456 and #457
Still progressing though failures in portlib
@DanHeidinga I see that you have a couple branches of stale changes for OSX. Would you like me to attempt to merge them and rebase?
Feel free. I haven't had a chance to get back to them so if you're able to, I'd appreciate the help.
After #738 is merged the current failure is within the port library looking like this
unix/j9sock.c:1287:13: error: implicit declaration of function 'gethostbyaddr_r' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
result = gethostbyaddr_r(addr,length,type, hostentBuffer, buffer, bufferSize, &herr);
^
unix/j9sock.c:1287:13: note: did you mean 'gethostbyaddr'?
/usr/include/netdb.h:270:17: note: 'gethostbyaddr' declared here
struct hostent *gethostbyaddr(const void *, socklen_t, int);
This needs some rework surrounding gethostby*_r.
Is the most recent OS X related work in this main repo? @DanHeidinga I noticed you have osx_asm
and osx_wip
branches, have those already been merged?
Anyone have a stab at build directions? I'm interested in pitching in, but want to start from the most forward point of progress.
Apologies to anyone following along as I've been remiss in updating this issue and especially to @brettwooldridge as I missed the offer of help.
There are two pending PRs with further fixes for mac (#2284 & #2274) from myself and @nbhuiyan.
Currently running into the following errors linking:
ld: warning: could not create compact unwind for _ffi_call_unix64: does not use RBP or RSP based frame
Undefined symbols for architecture x86_64:
"std::terminate()", referenced from:
___clang_call_terminate in libj9dyn.a(ROMClassBuilder.o)
___clang_call_terminate in libj9dyn.a(ROMClassWriter.o)
___clang_call_terminate in libj9dyn.a(SRPOffsetTable.o)
___clang_call_terminate in libj9dyn.a(ClassFileOracle.o)
___clang_call_terminate in libj9dyn.a(ClassFileParser.o)
___clang_call_terminate in libj9dyn.a(ConstantPoolMap.o)
"vtable for __cxxabiv1::__class_type_info", referenced from:
typeinfo for AllocationStrategy in libj9dyn.a(ROMClassBuilder.o)
typeinfo for Cursor in libj9dyn.a(ROMClassBuilder.o)
typeinfo for ClassFileOracle::AnnotationElementVisitor in libj9dyn.a(ROMClassWriter.o)
typeinfo for ConstantPoolMap::ConstantPoolVisitor in libj9dyn.a(ROMClassWriter.o)
typeinfo for ClassFileOracle::AnnotationsAttributeVisitor in libj9dyn.a(ROMClassWriter.o)
typeinfo for ClassFileOracle::AnnotationVisitor in libj9dyn.a(ROMClassWriter.o)
typeinfo for ClassFileOracle::ConstantPoolIndexVisitor in libj9dyn.a(ROMClassWriter.o)
...
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for __cxxabiv1::__si_class_type_info", referenced from:
typeinfo for SuppliedBufferAllocationStrategy in libj9dyn.a(ROMClassBuilder.o)
typeinfo for ROMClassWriter::AnnotationElementWriter in libj9dyn.a(ROMClassWriter.o)
typeinfo for ROMClassWriter::ConstantPoolWriter in libj9dyn.a(ROMClassWriter.o)
typeinfo for ROMClassWriter::ConstantPoolShapeDescriptionWriter in libj9dyn.a(ROMClassWriter.o)
typeinfo for ROMClassWriter::CallSiteWriter in libj9dyn.a(ROMClassWriter.o)
typeinfo for ClassFileOracle::InterfaceVisitor in libj9dyn.a(ClassFileOracle.o)
typeinfo for ComparingCursor in libj9dyn.a(ComparingCursor.o)
...
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for __cxxabiv1::__vmi_class_type_info", referenced from:
typeinfo for ROMClassWriter::AnnotationWriter in libj9dyn.a(ROMClassWriter.o)
typeinfo for ROMClassWriter::Helper in libj9dyn.a(ROMClassWriter.o)
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"___cxa_begin_catch", referenced from:
___clang_call_terminate in libj9dyn.a(ROMClassBuilder.o)
___clang_call_terminate in libj9dyn.a(ROMClassWriter.o)
___clang_call_terminate in libj9dyn.a(SRPOffsetTable.o)
___clang_call_terminate in libj9dyn.a(ClassFileOracle.o)
___clang_call_terminate in libj9dyn.a(ClassFileParser.o)
___clang_call_terminate in libj9dyn.a(ConstantPoolMap.o)
"___gxx_personality_v0", referenced from:
ROMClassBuilder::~ROMClassBuilder() in libj9dyn.a(ROMClassBuilder.o)
ROMClassBuilder::~ROMClassBuilder() in libj9dyn.a(ROMClassBuilder.o)
ROMClassBuilder::getROMClassBuilder(J9PortLibrary*, J9JavaVM*) in libj9dyn.a(ROMClassBuilder.o)
_shutdownROMClassBuilder in libj9dyn.a(ROMClassBuilder.o)
_j9bcutil_buildRomClassIntoBuffer in libj9dyn.a(ROMClassBuilder.o)
ROMClassBuilder::buildROMClass(ROMClassCreationContext*) in libj9dyn.a(ROMClassBuilder.o)
ROMClassBuilder::prepareAndLaydown(BufferManager*, ClassFileParser*, ROMClassCreationContext*) in libj9dyn.a(ROMClassBuilder.o)
...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Current errors are likely due to missing the -fno-strict-aliasing -fno-exceptions -fno-rtti -fno-threadsafe-statics
flags when compiling C++ files.
Current failure looks like:
/usr/bin/clang -shared -install_name libj9shr.dylib -Xlinker -rpath -Xlinker \$ORIGIN -m64 \
-L. -L../ -L../lib/ -o ../libj9shr29.dylib\
copyright.o shrclssup.o \
../lib/libj9util.a ../lib/libj9utilcore.a ../lib/libj9simplepool.a ../lib/libj9shrcommon.a ../lib/libj9shrutil.a ../lib/libj9zip.a -lj9avl -lj9hashtable -lj9pool ../libj9thr29.dylib ../libj9hookable29.dylib -g -lc -lm
Undefined symbols for architecture x86_64:
"___cxa_pure_virtual", referenced from:
vtable for SH_Manager in libj9shrcommon.a(Manager.o)
vtable for SH_OSCache in libj9shrcommon.a(OSCache.o)
vtable for SH_SharedCache in libj9shrcommon.a(CacheMap.o)
vtable for SH_OSCacheFile in libj9shrcommon.a(OSCacheFile.o)
vtable for SH_ROMClassResourceManager in libj9shrcommon.a(ROMClassResourceManager.o)
vtable for SH_TimestampManager in libj9shrcommon.a(TimestampManagerImpl.o)
ld: symbol(s) not found for architecture x86_64
Update: I tried compiling OpenJ9 Java 9 with clang/clang++ version 9.1.0.
1) warnings treated as errors: warnings_are_errors.log. Ran configure with --disable-warnings-as-errors
so that warnings aren't treated as errors. OpenJ9 warnings interpreted as errors would need to be fixed since we don't want to build OpenJ9 with --disable-warnings-as-errors
.
2) Encountered linkage error: Undefined symbols for architecture x86_64: "___cxa_pure_virtual", referenced from: ...
; Dan mentioned it above as well. I think we need to globally link the c++ library. In runtime/makelib/targets.mk.osx.inc.ftl:
UMA_EXE_POSTFIX_FLAGS += -lm -liconv -lc++ -lc -ldl -lutil -Wl,-rpath,\$$ORIGIN
3) With --disable-warnings-as-errors
and fix in 2), OpenJ9 compile completes. But, java executable has problems in loading libraries.
./build/macosx-x86_64-normal-server-release/jdk/bin/java -version
libjvm.so preloadLibrary(/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/libomrsig.dylib): dlopen(/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/libomrsig.dylib, 2): image not found
libomrsig failed to load: omrsig
preloadLibrary
assumes the library locations to be .../jdk/lib/
. But, the libraries are located in .../jdk/lib/compressedrefs
.
4) Hack to fix issue 3): Specify environment variable export DYLD_LIBRARY_PATH=.../jdk/lib/compressedrefs
.
5) Even after setting DYLD_LIBRARY_PATH
, libj9prt.dylib
can't be found. Even though, libj9prt.dylib
exists in .../jdk/lib/compressedrefs
.
./build/macosx-x86_64-normal-server-release/jdk/bin/java -version
libjvm.so preloadLibrary(openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/libj9vm29.dylib): dlopen(/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/libj9vm29.dylib, 2): Library not loaded: libj9prt.dylib
Referenced from: /openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/compressedrefs/libj9vm29.dylib
Reason: image not found
libjvm.so failed to load: j9vm29
6) Hacked issue 5) by changing library links in libj9vm29.dylib
using install_name_tool
.
Hack:
install_name_tool -change libj9prt.dylib /openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/compressedrefs/libj9prt29.dylib /openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/compressedrefs/libj9vm29.dylib
Before hack:
/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/compressedrefs/libj9vm29.dylib:
libj9vm.dylib (compatibility version 0.0.0, current version 0.0.0)
libj9thr29.dylib (compatibility version 0.0.0, current version 0.0.0)
libj9hookable29.dylib (compatibility version 0.0.0, current version 0.0.0)
libj9prt.dylib (compatibility version 0.0.0, current version 0.0.0)
libomrsig.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
After hack:
/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/compressedrefs/libj9vm29.dylib:
libj9vm.dylib (compatibility version 0.0.0, current version 0.0.0)
libj9thr29.dylib (compatibility version 0.0.0, current version 0.0.0)
libj9hookable29.dylib (compatibility version 0.0.0, current version 0.0.0)
/Users/sbabneet@ca.ibm.com/Documents/ibm_workspace/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/vm/libj9prt29.dylib (compatibility version 0.0.0, current version 0.0.0)
libomrsig.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
The proper way, to fix the library not found issue, is to include @rpath
(-install_name @rpath/lib$(UMA_TARGET_NAME).dylib
) when creating dylibs. When linking, rpath
needs to be properly defined -Wl,-rpath,@executable_path/../lib/compressedrefs
. @executable_path
points to the directory where java
executable is located. Using @executable_path
as a reference, we can locate JVM related dylibs. This won't help the preloadLibrary
issue in 3) since preloadLibrary
is given a wrong path.
7) After applying the hack in 6), java -version
worked without any library issues. Then, I encountered Abort trap: 6
. Using lldb
, i found that the abort
happens at statistics.c:44:
* thread #6, stop reason = signal SIGABRT
* frame #0: 0x00007fff5c042b6e libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff5c20d080 libsystem_pthread.dylib`pthread_kill + 333
frame #2: 0x00007fff5bf9e1ae libsystem_c.dylib`abort + 127
frame #3: 0x00007fff5bf9e321 libsystem_c.dylib`abort_report_np + 177
frame #4: 0x00007fff5bfc2bf5 libsystem_c.dylib`__chk_fail + 48
frame #5: 0x00007fff5bfc2bc5 libsystem_c.dylib`__chk_fail_overflow + 16
frame #6: 0x00007fff5bfc2e14 libsystem_c.dylib`__strcpy_chk + 83
frame #7: 0x0000000103150109 libj9vm29.dylib`addStatistic(javaVM=0x0000000103806c20, name="J9MemTag Version", dataType='\x06') at statistics.c:44
frame #8: 0x0000000103123c4f libj9vm29.dylib`protectedInitializeJavaVM(portLibrary=0x0000000100554408, userData=0x000070000d0aa660) at jvminit.c:5396
frame #9: 0x00000001033f621f libj9prt29.dylib`omrsig_protect(portLibrary=0x0000000100554408, fn=(libj9vm29.dylib`protectedInitializeJavaVM at jvminit.c:5318), fn_arg=0x000070000d0aa660, handler=(libj9vm29.dylib`structuredSignalHandlerVM at gphandle.c:593), handler_arg=0x0000000103806c20, flags=126, result=0x000070000d0aa650) at omrsignal.c:388
frame #10: 0x00000001031234fe libj9vm29.dylib`initializeJavaVM(osMainThread=0x0000000103803450, vmPtr=0x0000000100556860, createParams=0x000070000d0aa990) at jvminit.c:1000
frame #11: 0x00000001031194b4 libj9vm29.dylib`J9_CreateJavaVM(p_vm=0x0000000100556860, p_env=0x000070000d0aaeb8, createParams=0x000070000d0aa990) at jniinv.c:146
frame #12: 0x00000001005139e4 libjvm.dylib`JNI_CreateJavaVM(pvm=0x000070000d0aaec0, penv=0x000070000d0aaeb8, vm_args=0x000070000d0aae80) at jvm.c:2197
frame #13: 0x0000000100003a54 java`JavaMain + 267
frame #14: 0x00007fff5c20a661 libsystem_pthread.dylib`_pthread_body + 340
frame #15: 0x00007fff5c20a50d libsystem_pthread.dylib`_pthread_start + 377
frame #16: 0x00007fff5c209bf9 libsystem_pthread.dylib`thread_start + 13
statistics.c line 44 -- strcpy((char*)&(statistic->name), (char*)name)
-- i think we end up writing to memory that we don't own. i will try fixing this issue.
@DanHeidinga
i) for issue 2), do you agree that -lc++
should be globally linked?
ii) for issue 3), do we update the definition of preloadLibrary
or should the libraries be placed in .../jdk/lib/
?
awesome work @babsingh!
i) for issue 2), do you agree that -lc++ should be globally linked?
No, we don't want to link the c++ library if we can avoid it as user JNI code may require a different version of the c++ lib. For other platforms and components (jit), the c++ stdlib has been statically linked but we don't want to do that either as we don't want to statically link it more than once.
I had been investigating using -lc++abi
but hadn't looked at the license implications or the user JNI implications yet.
ii) for issue 3), do we update the definition of preloadLibrary or should the libraries be placed in .../jdk/lib/?
We can't move the libraries as it will break the fat packages that ship both default
and compressedrefs
in a single binary. We want the libs to stay in their package specific locations.
Why is preloadLibrary looking in the wrong place? How does that succeed on other platforms?
For issue 2 which component has these compilation issues? The GC defines a few functions like ___cxa_pure_virtual
so that we do not have to link with the c++ library I believe... maybe we just need to make this a more common thing? Or is it the GC and for some reason this is not working on OS X?
For issue 2 which component has these compilation issues?
___cxa_pure_virtual
issues were seen while building runtime/cfdumper
, runtime/bcutil/test/dyntest
and runtime/tests/shared
. In the below cases, ___cxa_pure_virtual
was used outside the scope of GC. I think we need to make ___cxa_pure_virtual
a more common thing.
cfdumper and bcutil/test/dyntest:
Undefined symbols for architecture x86_64:
"___cxa_pure_virtual", referenced from:
vtable for AllocationStrategy in libj9dyn.a(ROMClassBuilder.o)
vtable for ConstantPoolMap::ConstantPoolVisitor in libj9dyn.a(ROMClassWriter.o)
vtable for ClassFileOracle::AnnotationsAttributeVisitor in libj9dyn.a(ROMClassWriter.o)
vtable for ClassFileOracle::AnnotationVisitor in libj9dyn.a(ROMClassWriter.o)
vtable for ClassFileOracle::ConstantPoolIndexVisitor in libj9dyn.a(ROMClassWriter.o)
vtable for ClassFileOracle::ExceptionHandlerVisitor in libj9dyn.a(ROMClassWriter.o)
vtable for ClassFileOracle::StackMapFrameVisitor in libj9dyn.a(ROMClassWriter.o)
...
runtime/tests/shared:
Undefined symbols for architecture x86_64:
"___cxa_pure_virtual", referenced from:
vtable for AbstractMemoryPermission in ClassDebugDataTests.o
vtable for SH_OSCache::SH_OSCacheInitializer in CorruptCacheTest.o
vtable for SH_CompositeCache in CorruptCacheTest.o
vtable for AbstractMemoryPermission in CorruptCacheTest.o
vtable for SH_OSCache::SH_OSCacheInitializer in OSCacheTestMmap.o
vtable for SH_OSCache::SH_OSCacheInitializer in OSCacheTestSysv.o
vtable for SH_OSCache in libj9shrcommon.a(OSCache.o)
Why is preloadLibrary looking in the wrong place? How does that succeed on other platforms?
I will build on Linux to find differences in preloadLibrary
's functionality.
Update:
1) Fixed the abort
that happens at statistics.c:44.
2) libj9dmp29.dylib and libjclse9_29.dylib couldn't locate libj9zlib.dylib. Used install_name_tool to hack the path to libj9zlib.dylib within libj9dmp29.dylib and libjclse9_29.dylib.
3) I was able to get java -version
to run on macos. OpenJ9 was build without JIT so -Xint
had to be specified; otherwise, JVM attempted to load libj9jit29.dylib
.
./build/macosx-x86_64-normal-server-release/jdk/bin/java -version
JVMJ9VM011W Unable to load j9vrb29: dlopen(/openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/libj9vrb29.dylib, 2): Symbol not found: _get16BitByteCodeIndexFromExceptionTableVerbose Referenced from: /openj9-openjdk-jdk9/build/macosx-x86_64-normal-server-release/jdk/lib/compressedrefs/libj9vrb29.dylib Expected in: flat namespace in /openj9-openjdk-jdk9/build/macosx-x86_6
openjdk version "9.0.4-internal"
OpenJDK Runtime Environment (build 9.0.4-internal+0-adhoc.sbabneetcaibmcom.openj9-openjdk-jdk9)
Eclipse OpenJ9 VM (build macos_port_v1-acdf59a3, JRE 9 Darwin amd64-64-Bit 20180912_000000 (JIT disabled, AOT disabled)
OpenJ9 - acdf59a3
OMR - f29d1585
JCL - 4c213cb02d based on jdk-9.0.4+12)
4) There was another issue: Symbol not found: _get16BitByteCodeIndexFromExceptionTableVerbose
in libj9vrb29.dylib
. This didn't prevent the JVM to run. This symbol is defined in compiler/runtime/MethodMetaData.h:66
. I didn't compile with JIT. I think this issue would go away once OpenJ9 is compiled with JIT.
Since java -version
runs, I am guessing there won't be more issues that prevent OpenJ9 to build. I will start creating pull requests (fixes) for the issues found.
Update:
1) Recent fixes: #2858, #2929, #2985 and eclipse/omr#2966. This should fix majority of the issues mentioned above. The final remaining task is to fix how dylib libraries are named and linked.
2) I am using the following Clang compiler to compile OpenJ9 on OSX:
$ clang --version
Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
With the above Clang compiler, warnings are interpreted as errors for openj9-openjdk-jdk9 native source code: warning_as_errors_osx.txt. In AdoptOpenJDK Hotspot builds, clang C/C++ Apple LLVM version 7.0.2
is used for JDK11, and llvm-gcc-4.2/llvm-g++-4.2
is used for JDK8. In our tool chain, should we use the same compilers so that openjdk source compiles properly?
[Update] Fix RPATH and dylib install_name on OSX: https://github.com/eclipse/openj9/pull/3034, https://github.com/eclipse/omr/pull/2998.
There is another issue while loading dependent dylibs. For example,
otool -L libj9vm29.dylib
libj9vm29.dylib:
@rpath/libj9vm.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libj9thr29.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libj9hookable29.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libj9prt.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libomrsig.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
libj9vm29
depends on libj9prt
. The dylib file for libj9prt
is named libj9prt29.dylib
but the -install_name
is @rpath/libj9prt
. The dynamic linker tries to search for libj9prt.dylib
in the compressedrefs
directory. But, libj9prt.dylib
doesn't exist. After renaming libj9prt29.dylib
to libj9prt.dylib
, the linker finds the library. So, dylib file names should match the -install_name
. The same issue is seen for libj9zlib
.
I tried to change -install_name ...
to match the dylib file name. I found a lot of dependencies: UMA Java code, uma_macros.mk
, modules.xml
, j9lib.h.in
and j9lib.h.ftl
. I am not sure how to resolve this issue. @DanHeidinga what are your thoughts?
Update: 1) Fix for the above the issue (-install_name ... should match match the dylib file name): https://github.com/eclipse/openj9/pull/3056. 2) Unmerged PR - https://github.com/eclipse/openj9/pull/2985.
Once the above PRs are merged, OpenJ9-JDK9 can be built without a JIT.
Setup/Build Notes:
1) Changes to disable JIT in openj9-openjdk-jdk9 extensions repo.
2) Changes to disable JIT compilation in OpenJ9 repo.
3) gnu-tar is needed since bsdtar doesn't support --exclude-vcs
.
a. brew install gnu-tar
b. export TAR=gtar
4) JIT needs to be disabled during runtime: export IBM_JAVA_OPTIONS="-Xint"
.
5) Install freetype: brew install freetype
6) Download freemarker.jar:
wget https://sourceforge.net/projects/freemarker/files/freemarker/2.3.8/freemarker-2.3.8.tar.gz/download -O freemarker.tgz
tar -xzf freemarker.tgz freemarker-2.3.8/lib/freemarker.jar --strip=2
rm -f freemarker.tgz
7) Running configure:
./configure --with-freemarker-jar=<PATH_TO_FREEMARKER>/freemarker.jar --with-freetype=/usr/local/Cellar/freetype/2.9.1/ --disable-warnings-as-errors
8) make images
or make all
.
Working through these instructions to get an jdk8 env setup for mac builds: https://github.com/manasthakur/jdk-tips/wiki/Building-OpenJDK-8-on-Mac-OS-X-Yosemite
Hit the first road block on building with xcode 4.6.3: It's not supported on High Sierra.
Update: Java 8 OSX builds
a) Completed setup: installed 10.10 Yosemite and Xcode4 dev tools on an OSX machine.
b) Followed other setup instructions in https://github.com/manasthakur/jdk-tips/wiki/Building-OpenJDK-8-on-Mac-OS-X-Yosemite.
c) Got assembler errors:
.global
, it recognizes .globl
. stackoverflow post: https://stackoverflow.com/questions/10512248/unknown-pseudo-op-global. fpusup.s:125:Unknown pseudo-op: .global ... fpusup.s:125:Rest of line ignored. 1st junk character valued 95 (_).
ii) FATAL:/Applications/Xcode4/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../libexec/as/x86_64/as: I don't understand 'D' flag!. as -I. -I../include -I../oti -I../util -I../gc_include -I../omr/gc/include -I../compiler/runtime -I../nls -I../omr/include_core -DTR_HOST_X86 -DTR_64Bit -DTR_HOST_64BIT -DTR_TARGET_64BIT -DTR_TARGET_X86 -o xnathelp.o xnathelp.s
.export AS=/usr/bin/as
[Apple LLVM version 7.0.2 (clang-700.1.81)]
i) xnathelp.s:5595:Unknown pseudo-op: .extern ... xnathelp.s:5595:Rest of line ignored. 1st junk character valued 95 (_).
ii) xnathelp.s:9865:suffix or operands invalid for
jmp' **iii)**
xnathelp.s:10318:suffix or operands invalid for call'
iv) xnathelp.s:10524:Alignment too large: 15. assumed.
Possible options: 1) Update assembly syntax in .m4 files to match XCode assembler's syntax. 2) Use another assembler which is compatible with syntax in .m4 files.
Update: Java 9 compressedrefs
Updated make and spec files: a) openj9 repo - https://github.com/eclipse/openj9/pull/3144 b) JDK9 extensions repo - https://github.com/ibmruntimes/openj9-openjdk-jdk9/pull/179
Followed the below instructions for memory map support below 4GB on OSX:
a) https://github.com/eclipse/omr/issues/15
b) https://stackoverflow.com/questions/46916112/osx-ld-why-does-pagezero-size-default-to-4gb-on-64b-osx
c) ld: -pagezero_size option can only be used when linking a main executable. Updated flags.m4
: code_link. $2LDFLAGS_JDKEXE="${$2LDFLAGS_JDK} -pagezero_size 10000"
.
e) With the above option, got this error: JVMJ9VM013W Initialization error in function threadInitStages(4): cannot initialize VM threading
. Traced the above failure:
-- vmthinit.c::initializeVMThreading
---montable.c::initializeMonitorTable
----- montable.c::createMonitorTable
-------hashatable.c::hashTableNewImpl
----------pool.c::pool_new
.
The cause for
pool_new
's failure still needs to be investigated.
Without -pagezero_size
, GC fails to initialize with the below error:
JVMJ9VM015W Initialization error for library j9gc29(2): Failed to instantiate compressed references metadata; 200M requested
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Update: Java 9 with JIT enabled signal handling issue
=== (DivByZero.java) Testcase ===
public class DivByZero {
public int returnOne() {
try {
String str = null;
str.equals("a");
return 1;
} catch ( NullPointerException e1 ) {
try {
int intArray [] = new int [] {1};
return intArray[1];
} catch ( ArrayIndexOutOfBoundsException e2 ) {
return 1/0;
}
}
}
public static void main(String[] args) {
DivByZero db = new DivByZero();
db.returnOne();
}
}
=== Expected output ===
Exception in thread "main" java.lang.ArithmeticException: divide by zero
at DivByZero.returnOne(DivByZero.java:12)
at DivByZero.main(DivByZero.java:18)
[OpenJ9 on OSX] The above testcase works properly with -Xint
, -Xjit
or no extra JVM options. With Xjit:count=0
, OpenJ9 doesn't throw the ArithmeticException
on OSX. Also, the JVM hangs and doesn't terminate properly.
With -Xrs:sync -Xjit:count=0
, OpenJ9 runs the above testcase properly. -Xrs:sync
sets the following flags: J9_SIG_XRS_SYNC
(J9JavaVM->sigFlags) and OMRPORT_SIG_OPTIONS_REDUCED_SIGNALS_SYNCHRONOUS
(omr/port/unix/omrsignal.c::signalOptionsGlobal). This is another pending issue.
NASM version 2.13.03 required as per: https://github.com/eclipse/openj9/issues/2841#issuecomment-421044064
Reference - OSX 64-bit compressedrefs build JIT crash segfault - https://github.com/eclipse/openj9/issues/3223
Another issue - OpenJ9 OSX JDK doesn't work on older MacOS versions - https://github.com/eclipse/openj9/issues/3244
OpenJ9 JDK11 default/compressedrefs + JIT enabled runs on OSX. a) with signal handing (-Xrs) disabled, which affects performance b) https://github.com/eclipse/openj9/issues/3154
Update: OpenJ9 JDK8 on OSX
a) Machine setup: https://github.com/manasthakur/jdk-tips/wiki/Building-OpenJDK-8-on-Mac-OS-X-Yosemite
b) Unmerged fixes: https://github.com/eclipse/openj9/pull/3259, https://github.com/eclipse/omr/pull/3055.
c) Manual hack for xnathelp.s
errors --- remove .extern
statements --- then, continue make: cd ./build//macosx-x86_64-normal-server-release/vm/codert_vm/; cat xnathelp.s | grep -v "\.extern" > xnathelp.s; cd ../../../../; make LOG=trace images;
. Removing .extern
statements during *.m4
-> *.s
conversion broke something.
d) JIT was disabled. Xcode4 was used to built JDK8 on OSX. Xcode4 came with gcc version 4.2.1
. To build the JIT, gcc version 4.4.x+
is needed.
With a)-d), OpenJ9 compile completed
for OpenJ9 JDK8 on OSX. Then, I encountered an error related to CreateJars.gmk:658
(related to JDK tools). Will investigate further next week.
Update: OpenJ9 JDK8 on OSX
CreateJars.gmk:658
issue was resolved via https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/141. Afterwards, the following error was seen during JVM initialization:
java -Xint -version
JVMJ9VM015W Initialization error for library jclse7b_29(2): JVMJ9VM009E J9VMDllMain failed Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.
It will require brute-force debugging (print statements in initialization code) to triage the above error.
- Installing **gcc-4.9**:
sudo mkdir /Applications/Xcode7
sudo xcode-select -switch /Applications/Xcode7/Xcode.app
brew install gcc49
- **Xcode7 + gcc-4.9:**
- The below error related to `object.h ... dispatch_block_t` was seen. This is Clang code which is incompatible with GCC. **[HACK]** I fixed it by replacing `typedef void (^dispatch_block_t)(void);` with `typedef void* dispatch_block_t;` in `/usr/include/dispatch/object.h`. I also verified that https://github.com/eclipse/omr/pull/3055 is required. Also, I had to remove `-Werror=header-guard` from `openj9/runtime/compiler/build/toolcfg/gnu/common.mk`; compiler option is unrecognized. Afterwards, VM/GC/JIT compiled without any patches (reached `OpenJ9 compile complete` stage). https://github.com/eclipse/openj9/pull/3259 isn't required with **Xcode7 + gcc-4.9**. But, JDK source failed to compile with Xcode7.
In file included from /usr/include/dispatch/dispatch.h:51:0, from ../include_core/./unix/thrdsup.h:82, from ../include_core/thrdsup.h:35, from ./common/j9sem.c:32: /usr/include/dispatch/object.h:143:15: error: expected identifier or '(' before '^' token typedef void (^dispatch_block_t)(void);
- **Xcode 4 + gcc-4.9:**
- The below error related to `movq` was encountered. While installing **gcc-4.9**, `brew` had thrown an error that **gcc-4.9** is incompatible with **Xcode4**. Similar issue was seen elsewhere: https://github.com/libressl-portable/portable/issues/121 and https://github.com/SciRuby/nmatrix/issues/406. Couldn't reproduce the issue outside the makefiles; it wasn't even clear which command caused the below error; thus, couldn't fix the below error.
./tracemerge -majorversion 5 -minorversion 1 -root .
touch trace_merge
/var/folders/mp/8598ttz56rlgm_vhtcvs32gh0000gn/T//ccH3T1Pn.s:64912:suffix or operands invalid for movq' /var/folders/mp/8598ttz56rlgm_vhtcvs32gh0000gn/T//ccH3T1Pn.s:64922:suffix or operands invalid for
movq'
/var/folders/mp/8598ttz56rlgm_vhtcvs32gh0000gn/T//ccH3T1Pn.s:64953:suffix or operands invalid for movq' /var/folders/mp/8598ttz56rlgm_vhtcvs32gh0000gn/T//ccH3T1Pn.s:64956:suffix or operands invalid for
movq'
/var/folders/mp/8598ttz56rlgm_vhtcvs32gh0000gn/T//ccH3T1Pn.s:64958:suffix or operands invalid for movq' /var/folders/mp/8598ttz56rlgm_vhtcvs32gh0000gn/T//ccH3T1Pn.s:64978:suffix or operands invalid for
movq'
**Possible solution:** Build OpenJ9 (VM/JIT/GC/OMR) with **Xcode7 + gcc4.9** and build JDK source with **Xcode4 + gcc4.2.1**. @jdekonin is this possible? do you recollect the approach to achieve this?
@babsingh the manual way to try it, build with Xcode4 and then take OpenJ9 compiled with gcc4.9 and use it to replace the OpenJ9 compiled with gcc 4.2.1 and see how it runs.
Can both gcc 4.2.1 and gcc 4.9 be installed at the same time? May be able to set environment variables for the OpenJ9 part of the build to use gcc 4.9
With the existing linux systems we have more than 1 gcc installed and setup the environment at job time. JDK8 uses v4.8 whereas JDK11 uses v7.3. Setting CC or CXX and/or some sort of an environment setup script per release would be fine.
Example build_env vars, but there is also build_env cmds (just not used atm)
https://github.com/eclipse/openj9/blob/master/buildenv/jenkins/variables/defaults.yml
build_env cmds would execute once as the first action within the job.
@jdekonin Does that set the compiler for the whole OpenJDK build or just the OpenJ9 portion?
We're looking to use a newer compiler for just the OpenJ9 part and leave the OJDK part on the older compiler
@pshipton
the manual way ...
I put the default directory from a SDK built with Xcode7 + gcc4.9 into a SDK built with Xcode4 + gcc4.2.1. I could invoke java -version
... JIT loaded; but, got the same error as before:
JVMJ9VM015W Initialization error for library jclse7b_29(2): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Can both gcc 4.2.1 and gcc 4.9 be installed at the same time?
yes, both are installed at the same time. we need to dynamically change env vars: XCODE_PATH
, CC
, CXX
and others when building OpenJ9. restore the env var values after OpenJ9 is built. or we can just use custom env vars for building OpenJ9. i will investigate how to tweak the makefiles / build setup.
debug on :)
@DanHeidinga that is the entire environment. There isn't a way to split that with the existing scripting afaik. @vsebe or @keithc-ca, if either of you believe this can be done with the existing can you please comment?
@DanHeidinga @0xdaryl [Update] got JDK8 to work with JIT + compressedrefs
. creating pull requests now.
openjdk version "1.8.0_192-internal"
OpenJDK Runtime Environment (build 1.8.0_192-internal-jenkins_2018_10_17_11_24-b00)
Eclipse OpenJ9 VM (build master-2c817c52c, JRE 1.8.0 Mac OS X amd64-64-Bit Compressed References 20181017_000000 (JIT enabled, AOT enabled)
OpenJ9 - 2c817c52c
OMR - 4d96857a
JCL - fcd436bf56 based on jdk8u192-b03)
https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/147 (and equivalent PRs for jdk11 and jdk12) allow specification of distinct compilers for use in compiling OpenJ9.
As we've delivered mac on jdk 11 for the 0.11.0 release, moving the remaining work to 0.12.0
Mac support for jdk 8 & 11 is in 0.12.0
. Rolling this forward for tracking the other mac defects
Mac support for jdk 8, 11, 12 is in 0.14.0. Rolling this forward for tracking the other mac defects, although I'm not sure if this Issue has value any more. https://github.com/eclipse/openj9/projects/7 is also used for tracking remaining Mac issues. @DanHeidinga should we close this?
should we close this?
Yes, the project is sufficient for tracking mac-specific issues
It has been pointed out that Eclipse OpenJ9 does not yet have OS X builds. Eclipse OMR provides OS X support. In this issue we can discuss what would be needed.