Closed mickaelistria closed 2 years ago
DanHeidinga
commented
6 years ago $ java -Xjit:vmState=0x00040000
vmState [0x40000]: {J9VMSTATE_JNI}Shows this is a crash in JNI. Can you run with -Xcheck:jni which attempts to catch JNI errors at runtime?
mickaelistria
commented
6 years ago Sure
/home/mistria/sandbox/eclipse/features/org.eclipse.epp.runtime.openj9.feature_4.8.0.201806201617/jre/bin/java -Xcheck:jni -Dosgi.requiredJavaVersion=1.8 -Dosgi.instance.area.default=@user.home/eclipse-workspace -XX:+UseG1GC -XX:+UseStringDeduplication -Dosgi.requiredJavaVersion=1.8 -Dosgi.dataAreaRequiresExplicitInit=true -Xms256m -Xmx1024m -jar /home/mistria/sandbox/eclipse//plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar -os linux -ws gtk -arch x86_64 -showsplash /home/mistria/sandbox/eclipse//plugins/org.eclipse.epp.package.common_4.8.0.20180616-1200/splash.bmp -launcher /home/mistria/sandbox/eclipse/eclipse -name Eclipse --launcher.library /home/mistria/sandbox/eclipse//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.700.v20180518-1200/eclipse_1705.so -startup /home/mistria/sandbox/eclipse//plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar --launcher.appendVmargs -exitdata 47a8060 -product org.eclipse.epp.package.rust.product -vm /home/mistria/sandbox/eclipse/features/org.eclipse.epp.runtime.openj9.feature_4.8.0.201806201617/jre/bin/java -vmargs -Xcheck:jni -Dosgi.requiredJavaVersion=1.8 -Dosgi.instance.area.default=@user.home/eclipse-workspace -XX:+UseG1GC -XX:+UseStringDeduplication -Dosgi.requiredJavaVersion=1.8 -Dosgi.dataAreaRequiresExplicitInit=true -Xms256m -Xmx1024m -jar /home/mistria/sandbox/eclipse//plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar(see addition of -Xcheck:jni) shows
JVMJNCK001I JNI check utility installed. Use -Xcheck:jni:help for usage
Unhandled exception
Type=Segmentation error vmState=0x00040000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F3B969625D0 Handler2=00007F3B95ABD330 InaccessibleAddress=0000000000007210
RDI=000000000000002F RSI=00007FFDCC0F33D8 RAX=0000000000000001 RBX=00007F3B291CD680
RCX=00007F3B90521A00 RDX=00007F3B90521A00 R8=00007F3B90524740 R9=000000000000002F
R10=00007F3B903ACEA0 R11=00007F3B9005AD58 R12=00007FFDCC0F33D8 R13=00007F3B90521A00
R14=00007F3B291CD690 R15=0000000000000000
RIP=0000000000007210 GS=0000 FS=0000 RSP=00007F3B978B4C98
EFlags=0000000000010202 CS=0033 RBP=000000000000002F ERR=0000000000000014
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000007210
xmm0 00007f3b986ed3e8 (f: 2557400064.000000, d: 6.911679e-310)
xmm1 00007f3b90059108 (f: 2416283904.000000, d: 6.911672e-310)
xmm2 6369746174735f72 (f: 1953718144.000000, d: 7.685180e+170)
xmm3 6369746174735f72 (f: 1953718144.000000, d: 7.685180e+170)
xmm4 656e5f6c616e6769 (f: 1634625408.000000, d: 3.938506e+180)
xmm5 00007f3b90527880 (f: 2421323776.000000, d: 6.911672e-310)
xmm6 00007f3b986ec990 (f: 2557397504.000000, d: 6.911679e-310)
xmm7 00007f3b90524c50 (f: 2421312512.000000, d: 6.911672e-310)
xmm8 00007f3b900b86d0 (f: 2416674560.000000, d: 6.911672e-310)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 3bbcc86800000000 (f: 0.000000, d: 6.095003e-21)
xmm11 3c76541148cbb8a2 (f: 1221310592.000000, d: 1.936679e-17)
xmm12 bfd5be02466294dc (f: 1180865792.000000, d: -3.397222e-01)
xmm13 3c72c00000000000 (f: 0.000000, d: 1.626303e-17)
xmm14 bbc396ecc5eda344 (f: 3320685312.000000, d: -8.296497e-21)
xmm15 4033687a9f1af100 (f: 2669342976.000000, d: 1.940812e+01)
Target=2_90_20180315_120 (Linux 4.16.13-300.fc28.x86_64)
CPU=amd64 (4 logical CPUs) (0x2db351000 RAM)
----------- Stack Backtrace -----------
(0x00007F3B95AD6BD2 [libj9prt29.so+0x38bd2])
(0x00007F3B95ABDF13 [libj9prt29.so+0x1ff13])
(0x00007F3B95AD6C4C [libj9prt29.so+0x38c4c])
(0x00007F3B95AD6D56 [libj9prt29.so+0x38d56])
(0x00007F3B95ABDF13 [libj9prt29.so+0x1ff13])
(0x00007F3B95AD6AA1 [libj9prt29.so+0x38aa1])
(0x00007F3B96962103 [libj9vm29.so+0x9e103])
(0x00007F3B95ABDF13 [libj9prt29.so+0x1ff13])
(0x00007F3B96962350 [libj9vm29.so+0x9e350])
(0x00007F3B95ABD531 [libj9prt29.so+0x1f531])
(0x00007F3B97ADEFC0 [libpthread.so.0+0x11fc0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2018/06/21 06:47:22 - please wait.
JVMDUMP032I JVM requested System dump using '/home/mistria/sandbox/eclipse/core.20180621.064722.3429.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.3451.
JVMDUMP012E Error in System dump: The core file created by child process with pid = 3451 was not found. Expected to find core file with name "/home/mistria/sandbox/eclipse/core.3451"
JVMDUMP032I JVM requested Java dump using '/home/mistria/sandbox/eclipse/javacore.20180621.064722.3429.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/mistria/sandbox/eclipse/javacore.20180621.064722.3429.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/mistria/sandbox/eclipse/Snap.20180621.064722.3429.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/mistria/sandbox/eclipse/Snap.20180621.064722.3429.0003.trc
JVMDUMP007I JVM Requesting JIT dump using '/home/mistria/sandbox/eclipse/jitdump.20180621.064722.3429.0004.dmp'
JVMDUMP010I JIT dump written to /home/mistria/sandbox/eclipse/jitdump.20180621.064722.3429.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".
JVMDUMP039I Processing dump event "abort", detail "" at 2018/06/21 06:48:20 - please wait.
JVMDUMP032I JVM requested System dump using '/home/mistria/sandbox/eclipse/core.20180621.064820.3429.0005.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.3504.
JVMDUMP012E Error in System dump: The core file created by child process with pid = 3504 was not found. Expected to find core file with name "/home/mistria/sandbox/eclipse/core.3504"
JVMDUMP032I JVM requested Java dump using '/home/mistria/sandbox/eclipse/javacore.20180621.064820.3429.0006.txt' in response to an event
JVMDUMP010I Java dump written to /home/mistria/sandbox/eclipse/javacore.20180621.064820.3429.0006.txt
JVMDUMP032I JVM requested Snap dump using '/home/mistria/sandbox/eclipse/Snap.20180621.064820.3429.0007.trc' in response to an event
JVMDUMP010I Snap dump written to /home/mistria/sandbox/eclipse/Snap.20180621.064820.3429.0007.trc
JVMDUMP007I JVM Requesting JIT dump using '/home/mistria/sandbox/eclipse/jitdump.20180621.064820.3429.0008.dmp'
JVMDUMP010I JIT dump written to /home/mistria/sandbox/eclipse/jitdump.20180621.064820.3429.0008.dmp
JVMDUMP013I Processed dump event "abort", detail "".I do keep the various dump files here. Would you like me to share some of them?
mickaelistria
commented
6 years ago Forgot to mention, but I do see the (exact?) same thing with all versions of OpenJDK+OpenJ9:
mickaelistria
commented
6 years ago Just some context, Eclipse IDE heavily relies on JNI for SWT and binds to various libs on Linux. The main ones would be libgtk, libcairo and libwebkitgtk IIRC.
DanHeidinga
commented
6 years ago Would you like me to share some of them?
That would be a big help.
mickaelistria
commented
6 years ago There are some debug file: https://gist.github.com/mickaelistria/0d2205ae30ad6d8a46c785fabdf599a7 Do you also need the .trc file? If yes, what's the preferred way to share this file (since it's binary, Gist aren't great)?
DanHeidinga
commented
6 years ago The javacore shows that the crash is occurring on the main thread in JNIBridge._show_splash native method. The c stdlib is raising an abort: abort+0x12b (0x00007FD794428561 [libc.so.6+0x21561])
1XMCURTHDINFO Current thread
3XMTHREADINFO "main" J9VMThread:0x0000000000FC0700, omrthread_t:0x00007FD78C008560, java/lang/Thread:0x00000000FC004618, state:R, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)
3XMTHREADINFO1 (native thread ID:0x1597, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)
3XMTHREADINFO2 (native stack address range from:0x00007FD7935CE000, to:0x00007FD793DCE000, size:0x800000)
3XMCPUTIME CPU usage total: 0.332300874 secs, current category="Application"
3XMHEAPALLOC Heap bytes allocated since last GC cycle=2053288 (0x1F54A8)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at org/eclipse/equinox/launcher/JNIBridge._show_splash(Native Method)
4XESTACKTRACE at org/eclipse/equinox/launcher/JNIBridge.showSplash(JNIBridge.java:112)
4XESTACKTRACE at org/eclipse/equinox/launcher/Main.handleSplash(Main.java:2197)
4XESTACKTRACE at org/eclipse/equinox/launcher/Main.basicRun(Main.java:589)
4XESTACKTRACE at org/eclipse/equinox/launcher/Main.run(Main.java:1498)
4XESTACKTRACE at org/eclipse/equinox/launcher/Main.main(Main.java:1471)
3XMTHREADINFO3 Native callstack:
<snip>
4XENATIVESTACK gsignal+0x10b (0x00007FD79443DF2B [libc.so.6+0x36f2b])
4XENATIVESTACK abort+0x12b (0x00007FD794428561 [libc.so.6+0x21561])
4XENATIVESTACK (0x00007FD791FD45BE [libj9prt29.so+0x1f5be])
4XENATIVESTACK (0x00007FD793FF5FC0 [libpthread.so.0+0x11fc0])So I could simply pass the -nosplash setting to Eclipse IDE launcher to avoid the splash and see how it proceeds. The process went a bit further and I see no segfault, but then Eclipse IDE fails just after with
!SESSION 2018-06-22 20:11:58.006 -----------------------------------------------
eclipse.buildId=4.8.0.I20180611-0500
java.fullversion=10.0.1+10
JRE 10 Linux amd64-64-Bit Compressed References 20180619_82 (JIT enabled, AOT enabled)
OpenJ9 - 3926aa29
OMR - 12fd129d
JCL - 3e42389178 based on jdk-10.0.1+10
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -product org.eclipse.epp.package.rust.product
Command-line arguments: -os linux -ws gtk -arch x86_64 -product org.eclipse.epp.package.rust.product
!ENTRY org.eclipse.osgi 4 0 2018-06-22 20:12:00.132
!MESSAGE Application error
!STACK 1
java.lang.UnsatisfiedLinkError: org/eclipse/swt/internal/gtk/OS._cachejvmptr()V
at org.eclipse.swt.internal.gtk.OS.cachejvmptr(OS.java:481)
at org.eclipse.swt.internal.gtk.OS.<clinit>(OS.java:92)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:134)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:80)
at org.eclipse.swt.widgets.Display.<clinit>(Display.java:142)
at org.eclipse.ui.internal.Workbench.createDisplay(Workbench.java:772)
at org.eclipse.ui.PlatformUI.createDisplay(PlatformUI.java:160)
at org.eclipse.ui.internal.ide.application.IDEApplication.createDisplay(IDEApplication.java:182)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:125)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@10.0.1/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@10.0.1/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@10.0.1/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@10.0.1/Method.java:564)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:656)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:592)
at org.eclipse.equinox.launcher.Main.run(Main.java:1498)
at org.eclipse.equinox.launcher.Main.main(Main.java:1471)It also seems like a JNI issue, just another form of it.
DanHeidinga
commented
6 years ago The UnsatisfiedLinkError looks like a native library is missing from the library path.
Can you share recreate steps or a docker image that reproduces the issue?
mickaelistria
commented
6 years ago The UnsatisfiedLinkError looks like a native library is missing from the library path.
Does OpenJ9 use another definition of library path than the default one HotSpot uses? Because with hotspot, I get proper linking, with OpenJ9 it's different, so it could just be that the default library path differ or is used. Or maybe Eclipse is actually dynamically modifying the library path and OpenJ9 doesn't allow it?
Can you share recreate steps or a docker image that reproduces the issue?
Sure, I'm using Fedora 28.
eclipse.ini, add the 2 following lines before the -vmargs line (with your favorite local path
-vm
/path/to/openj9/jdk-10.0.1+10/bin./eclipseeclipse.ini, replace the 2 lines
-showsplash
org.eclipse.epp.package.commonwith the 1 line
-nosplashconfiguration/<timestamp>.log file which shows the log with UnsatisfiedLinkError.mickaelistria
commented
6 years ago In some other circumstances (running the IDE from inside the IDE), I can see this kind of messages
!SESSION 2018-06-22 21:50:26.214 -----------------------------------------------
eclipse.buildId=4.8.0.I20180531-0700
java.fullversion=10.0.1+10
JRE 10 Linux amd64-64-Bit Compressed References 20180619_82 (JIT enabled, AOT enabled)
OpenJ9 - 3926aa29
OMR - 12fd129d
JCL - 3e42389178 based on jdk-10.0.1+10
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -product org.eclipse.platform.ide
Command-line arguments: -product org.eclipse.platform.ide -data /home/mistria/workspace/../runtime-parallelBuildsAcuteJDT -dev file:/home/mistria/workspace/.metadata/.plugins/org.eclipse.pde.core/Parallel Builds JDT/dev.properties -os linux -ws gtk -arch x86_64 -consoleLog
!ENTRY org.eclipse.osgi 4 0 2018-06-22 21:53:42.970
!MESSAGE Application error
!STACK 1
java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons:
/home/mistria/workspace/.metadata/.plugins/org.eclipse.pde.core/Parallel Builds JDT/org.eclipse.osgi/595/0/.cp/libswt-pi-gtk-4880.so (/lib64/libharfbuzz.so.0: undefined symbol: FT_Get_Var_Blend_Coordinates)
swt-pi-gtk (Not found in java.library.path)
/home/mistria/.swt/lib/linux/x86_64/libswt-pi-gtk-4880.so (/lib64/libharfbuzz.so.0: undefined symbol: FT_Get_Var_Blend_Coordinates)
/home/mistria/.swt/lib/linux/x86_64/libswt-pi-gtk.so (/home/mistria/.swt/lib/linux/x86_64/liblibswt-pi-gtk.so.so: cannot open shared object file: No such file or directory)
at org.eclipse.swt.internal.Library.loadLibrary(Library.java:327)
at org.eclipse.swt.internal.Library.loadLibrary(Library.java:239)
at org.eclipse.swt.internal.gtk.OS.<clinit>(OS.java:89)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:134)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Converter.java:80)
at org.eclipse.swt.widgets.Display.<clinit>(Display.java:142)
at org.eclipse.ui.internal.Workbench.createDisplay(Workbench.java:772)
at org.eclipse.ui.PlatformUI.createDisplay(PlatformUI.java:160)which I do not get with HotSpot. This seems to support my previous idea that there might be a difference between the default library path of OpenJ9 vs Hotspot.
mickaelistria
commented
6 years ago Here are the java.library.path I get at runtime (placing a breakpoint a bit before the UnsatisfiedLinkError)
/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib/home/mistria/git/org.eclipse.epp.packages/runtime/unit/target/jdk-10.0.1+10/lib/compressedrefs:/home/mistria/git/org.eclipse.epp.packages/runtime/unit/target/jdk-10.0.1+10/lib:/usr/lib64:/usr/libWhile debugging the OpenJ9-based app, I changed the java.library.path to get closer from the one with HotSpot, setting it to /home/mistria/git/org.eclipse.epp.packages/runtime/unit/target/jdk-10.0.1+10/lib/compressedrefs:/home/mistria/git/org.eclipse.epp.packages/runtime/unit/target/jdk-10.0.1+10/lib:/usr/lib64:/lib64:/lib:/usr/lib, and got the same result.
Looking at the WWW for hints, it seems like symbol FT_Get_Var_Blend_Coordinates come from libfreetype. And I noticed that OpenJ9 provides in it's lib folder a version of libfreetype that's different from the one on my system (both are called libfreetype.so.6 but they have different signature, and the one on my machine is actually a link to libfreetype.so.6.14.0).
I think this could be the issue: the libfreetype.so shipped by OpenJ9 may not be compatible with the libharfbuzz.so.0 that SWT and Eclipse IDE would like to load.
I'll try some more experiments on this topic and will report.
mickaelistria
commented
6 years ago Ok, I can confirm that removing libfreetype.so from the lib folder of jdk-10.0.1+10 allows to avoid both the Segfault and UnsatisfiedLinkError.
Now, the question is could OpenJ9 not ship libfreetype for its Linux package, as all distros are likely to have it installed?
pshipton
commented
6 years ago libfreetype*.so is something AdoptOpenJDK is adding to the builds (its not there when building OpenJDK + OpenJ9 via the instructions at the OpenJ9 website). Do you get the same behavior using a HotSpot build from AdoptOpenJDK (which also includes libfreetype.so)?
mickaelistria
commented
6 years ago Do you get the same behavior using a HotSpot build from AdoptOpenJDK (which also includes libfreetype.so)?
Indeed, I get exactly the same issues (just the formatting of the segfault report is different), and the same workaround of removing libfreetype.so from the lib/ folder fixes those issues. So does that mean I should report this issue to AdoptOpenJDK ?
I'm using the Fedora packages which are another build of openJDK (not based on AdoptOpenJDK AFAIK). Actually, I'm not sure the AdoptOpenJDK builds are the best candidates for inclusion in Eclipse IDE packages. Are there other builds generally available that I could try?
pshipton
commented
6 years ago Please do report the libfreetype.so issue at AdoptOpenJDK https://github.com/AdoptOpenJDK/openjdk-build/issues
Is there another problem with the AdoptOpenJDK builds besides the freetype one? AdoptOpenJDK are the official builds for OpenJ9, intended to be the place for getting tested builds for both OpenJDK and OpenJ9. You can build your own using the instructions [1], or snag an OpenJ9 nightly build artifact (except there are no builds for the release branches), but there would be issues such as not having any security certificates for Java 8.
mickaelistria
commented
6 years ago
Environment: Fedora 28 Using: https://github.com/AdoptOpenJDK/openjdk9-openj9-nightly/releases/download/jdk-9.0.4%2B12-201819061047/OpenJDK9-OPENJ9_x64_Linux_201819061047.tar.gz Command:
Where
/home/mistria/sandbox/eclipse/features/org.eclipse.epp.runtime.openj9.feature_4.8.0.201806201617/jre/bin/javais the java binary from package linked aboveExpected: It all works ;) Got: