search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

OpenSSL v3 causes abort on mac? #17577

Closed pshipton closed 1 year ago

pshipton commented 1 year ago

In https://github.com/eclipse-openj9/openj9/pull/14900 I did jdk11 builds for all platforms using OpenSSL 3.09. There was an "abort" event on the two mac platforms, which I assume is related to OpenSSL.

https://openj9-jenkins.osuosl.org/job/Build_JDK11_x86-64_mac_Personal/513 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Build_JDK11_x86-64_mac_Personal/513/Build_JDK11_x86-64_mac_Personal-513-20230531-175153-diagnostics.tar.gz

https://openj9-jenkins.osuosl.org/job/Build_JDK11_aarch64_mac_Personal/342 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Build_JDK11_aarch64_mac_Personal/342/Build_JDK11_aarch64_mac_Personal-342-20230601-074019-diagnostics.tar.gz

On amac there is a warning related to libcrypto. I expect you could get a native stack trace for the "abort" by loading the core file into the debugger.

17:40:03  WARNING: /Users/jenkins/workspace/Build_JDK11_aarch64_mac_Personal/build/macosx-aarch64-normal-server-release/jdk/bin/jmod is loading libcrypto in an unsafe way
17:40:03  JVMDUMP039I Processing dump event "abort", detail "" at 2023/06/01 07:40:02 - please wait.

@jasonkatonica

jasonkatonica commented 1 year ago

I looked into the core file core.20230601.074002.43574.0001.dmp which was within the file downloaded above named Build_JDK11_aarch64_mac_Personal-342-20230601-074019-diagnostics.tar.gz

> jdmpview -core core.20230601.074002.43574.0001.dmp -verbose
Console logging is now enabled for com.ibm.jvm.dtfjview.logger
Console logging is now enabled for com.ibm.dtfj.log
Startup parameters:
    -core = core.20230601.074002.43574.0001.dmp
    -verbose = true
Batch mode commands: []
DTFJView version 4.29.5, using DTFJ version 1.12.29003
Loading image from DTFJ...

Jun 12, 2023 4:34:33 PM com.ibm.jvm.dtfjview.commands.OpenCommand imagesFromCommandLine
FINE: Time taken to load image 847575459 ns
Jun 12, 2023 4:34:33 PM com.ibm.jvm.dtfjview.Session imageFromCommandLine
FINE: Time taken to load image 1244930917 ns
For a list of commands, type "help"; for how to use "help", type "help help"
Available contexts (* = currently selected context) :

Source : file:///Users/jasonkatonica/Downloads/Build_JDK11_aarch64_mac_Personal-342-20230601-074019-diagnostics/make/core.20230601.074002.43574.0001.dmp
    *0 : PID: 0 : JRE 11 Mac OS X aarch64-64 (build 11.0.20-internal+0-adhoc.jenkins.BuildJDK11aarch64macPersonal)

Then I ran the command as so to see the information about the current thread ( ID 10A5FDD88 ):

> info thread
 process id: 43574

 no native threads found with specified id

Java threads not associated with known native threads:

    name:          main
    id:            37958794
    Thread object: java/lang/Thread @ 0x10a5fdd88
    Native info:   !j9vmthread 0x12b00af00  !stack 0x12b00af00
    Daemon:        false
    ID:            1 (0x1)
    Priority:      5
    Thread.State:  RUNNABLE
    JVMTI state:   ALIVE RUNNABLE
    Java stack frames:
     bp: 0x000000014a815708  method: long jdk/crypto/jniprovider/NativeCrypto.loadCrypto(boolean)  (Native Method)
      objects: <no objects in this frame>
     bp: 0x000000014a815738  method: long jdk/crypto/jniprovider/NativeCrypto.loadCryptoLibraries()  source: NativeCrypto.java:79
      objects: <no objects in this frame>
     bp: 0x000000014a815750  method: Long jdk/crypto/jniprovider/NativeCrypto.lambda$new$0()  source: NativeCrypto.java:97
      objects: <no objects in this frame>
     bp: 0x000000014a815770  method: Object jdk/crypto/jniprovider/NativeCrypto$$Lambda$96/0x0000000000000000.run()  source: null:<data unavailable>Jun 12, 2023 4:34:39 PM com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand printJavaStackFrameInfo
FINE: <data unavailable>
com.ibm.dtfj.image.DataUnavailable
    at com.ibm.j9ddr.vm29.view.dtfj.java.DTFJJavaStackFrame.getLineNumber(DTFJJavaStackFrame.java:136)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.printJavaStackFrameInfo(InfoThreadCommand.java:775)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.printJavaThreadInfo(InfoThreadCommand.java:603)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.printAddressSpaceInfo(InfoThreadCommand.java:171)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.doCommand(InfoThreadCommand.java:145)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.run(InfoThreadCommand.java:99)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.tryCommand(Context.java:141)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.execute(Context.java:97)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:173)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:111)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.SimpleRedirectorCommand.run(SimpleRedirectorCommand.java:58)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.tryCommand(Context.java:141)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.execute(Context.java:97)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:173)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.execute(Session.java:813)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.execute(Session.java:767)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.executeJdmpviewCommand(ToolsRegistry.java:183)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.execute(ToolsRegistry.java:107)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.execute(ToolsRegistry.java:88)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.recordAndExecute(ToolsRegistry.java:72)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.runInteractive(Session.java:741)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.run(Session.java:665)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.launch(DTFJView.java:52)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.main(DTFJView.java:46)

      objects: <no objects in this frame>
     bp: 0x000000014a815790  method: Object java/security/AccessController.doPrivileged(java.security.PrivilegedAction)  source: AccessController.java:690
      objects: <no objects in this frame>
     bp: 0x000000014a8157b8  method: void jdk/crypto/jniprovider/NativeCrypto.<init>()  source: NativeCrypto.java:97
      objects: 0x129ece268
     bp: 0x000000014a8157d8  method: void jdk/crypto/jniprovider/NativeCrypto$InstanceHolder.<clinit>()  source: NativeCrypto.java:64
      objects: 0x129ece268
     bp: 0x000000014a815840  method: long jdk/crypto/jniprovider/NativeCrypto.getVersionIfAvailable()  source: NativeCrypto.java:119
      objects: <no objects in this frame>
     bp: 0x000000014a815858  method: boolean jdk/crypto/jniprovider/NativeCrypto.isAllowedAndLoaded()  source: NativeCrypto.java:108
      objects: <no objects in this frame>
     bp: 0x000000014a8158d8  method: void sun/security/provider/SunEntries.<init>(java.security.Provider)  source: SunEntries.java:226
      objects: 0x129ec9d38 0x129ec83e0 0x129ec9dc0
     bp: 0x000000014a815910  method: void sun/security/provider/Sun.<init>()  source: Sun.java:54
      objects: 0x129ec83e0 0x129ec9d38
     bp: 0x000000014a815950  method: java.security.Provider sun/security/jca/ProviderConfig.getProvider()  source: ProviderConfig.java:189
      objects: 0x129ec72d0 0x129ec72d0 0x129ec83e0
     bp: 0x000000014a815980  method: java.security.Provider sun/security/jca/ProviderList.getProvider(int)  source: ProviderList.java:290
      objects: <no objects in this frame>
     bp: 0x000000014a8159d0  method: java.security.Provider$Service sun/security/jca/ProviderList.getService(String, String)  source: ProviderList.java:404
      objects: 0x129ec7170 0x10a7997b0 0x10a799238
     bp: 0x000000014a815a30  method: sun.security.jca.GetInstance$Instance sun/security/jca/GetInstance.getInstance(String, Class, String)  source: GetInstance.java:157
      objects: 0x10a7997b0 0x10a799570 0x10a799238 0x129ec7170
     bp: 0x000000014a815a60  method: Object[] java/security/Security.getImpl(String, String, String)  source: Security.java:716
      objects: <no objects in this frame>
     bp: 0x000000014a815a90  method: java.security.MessageDigest java/security/MessageDigest.getInstance(String)  source: MessageDigest.java:178
      objects: 0x10a799238
     bp: 0x000000014a815ac8  method: byte[] jdk/internal/module/ModuleHashes.computeHash(module.ModuleReader, String)  source: ModuleHashes.java:113
      objects: 0x129eb5d50
     bp: 0x000000014a815b20  method: jdk.internal.module.ModuleHashes jdk/internal/module/ModuleHashes.generate(java.util.Set, String)  source: ModuleHashes.java:167
      objects: 0x10a799238 0x129eb5b48 0x129eb5bb0 0x12a3f8028 0x129eb5d50
     bp: 0x000000014a815b80  method: void jdk/internal/module/ModuleHashesBuilder.lambda$computeHashes$3(jdk.internal.module.ModuleHashesBuilder$Graph, java.util.Set, java.util.Map, String)  source: ModuleHashesBuilder.java:120
      objects: 0x129e71758 0x12a3da770
     bp: 0x000000014a815c18  method: void jdk/internal/module/ModuleHashesBuilder$$Lambda$89/0x0000000000000000.accept(Object)  source: null:<data unavailable>Jun 12, 2023 4:34:39 PM com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand printJavaStackFrameInfo
FINE: <data unavailable>
com.ibm.dtfj.image.DataUnavailable
    at com.ibm.j9ddr.vm29.view.dtfj.java.DTFJJavaStackFrame.getLineNumber(DTFJJavaStackFrame.java:136)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.printJavaStackFrameInfo(InfoThreadCommand.java:775)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.printJavaThreadInfo(InfoThreadCommand.java:603)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.printAddressSpaceInfo(InfoThreadCommand.java:171)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.doCommand(InfoThreadCommand.java:145)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.infocommands.InfoThreadCommand.run(InfoThreadCommand.java:99)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.tryCommand(Context.java:141)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.execute(Context.java:97)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:173)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:111)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.commands.SimpleRedirectorCommand.run(SimpleRedirectorCommand.java:58)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.tryCommand(Context.java:141)
    at openj9.dtfj/com.ibm.java.diagnostics.utils.Context.execute(Context.java:97)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:173)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.execute(Session.java:813)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.execute(Session.java:767)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.executeJdmpviewCommand(ToolsRegistry.java:183)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.execute(ToolsRegistry.java:107)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.execute(ToolsRegistry.java:88)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.recordAndExecute(ToolsRegistry.java:72)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.runInteractive(Session.java:741)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.run(Session.java:665)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.launch(DTFJView.java:52)
    at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.main(DTFJView.java:46)

      objects: <no objects in this frame>
     bp: 0x000000014a815c58  method: void java/util/stream/ForEachOps$ForEachOp$OfRef.accept(Object)  source: ForEachOps.java:183
      objects: 0x12a3da770 0x129e8e948 0x129e8e918
     bp: 0x000000014a815cc8  method: void java/util/stream/ReferencePipeline$2$1.accept(Object)  source: ReferencePipeline.java:177
      objects: 0x12a3da770 0x129e8e968 0x129e86878
     bp: 0x000000014a815d88  method: void java/util/ArrayDeque$DeqSpliterator.forEachRemaining(java.util.function.Consumer)  source: ArrayDeque.java:849
      objects: 0x129e8e968 0x129e80428
     bp: 0x000000014a815df0  method: void java/util/stream/AbstractPipeline.copyInto(java.util.stream.Sink, java.util.Spliterator)  source: AbstractPipeline.java:497
      objects: 0x129e80928 0x129e8e968 0x129e86898 0x129e8e948 0x10a603778
     bp: 0x000000014a815e20  method: java.util.stream.Sink java/util/stream/AbstractPipeline.wrapAndCopyInto(java.util.stream.Sink, java.util.Spliterator)  source: AbstractPipeline.java:487
      objects: 0x129e8e948
     bp: 0x000000014a815e50  method: Void java/util/stream/ForEachOps$ForEachOp.evaluateSequential(java.util.stream.PipelineHelper, java.util.Spliterator)  source: ForEachOps.java:150
      objects: <no objects in this frame>
     bp: 0x000000014a815ef0  method: Object java/util/stream/ForEachOps$ForEachOp$OfRef.evaluateSequential(java.util.stream.PipelineHelper, java.util.Spliterator)  source: ForEachOps.java:173
      objects: <no objects in this frame>
     bp: 0x000000014a815f38  method: Object java/util/stream/AbstractPipeline.evaluate(java.util.stream.TerminalOp)  source: AbstractPipeline.java:239
      objects: 0x129e86898
     bp: 0x000000014a815f68  method: void java/util/stream/ReferencePipeline.forEach(java.util.function.Consumer)  source: ReferencePipeline.java:497
      objects: <no objects in this frame>
     bp: 0x000000014a815fc8  method: java.util.Map jdk/internal/module/ModuleHashesBuilder.computeHashes(java.util.Set)  source: ModuleHashesBuilder.java:104
      objects: 0x129e71758
     bp: 0x000000014a815fe8  method: java.util.Map jdk/tools/jmod/JmodTask$Hasher.computeHashes()  source: JmodTask.java:926
      objects: <no objects in this frame>
     bp: 0x000000014a816030  method: jdk.internal.module.ModuleHashes jdk/tools/jmod/JmodTask$JmodFileWriter.computeHashes(module.ModuleDescriptor)  source: JmodTask.java:654
      objects: 0x12a3da770
     bp: 0x000000014a8160a0  method: void jdk/tools/jmod/JmodTask$JmodFileWriter.writeModuleInfo(jdk.tools.jmod.JmodOutputStream, java.util.Set)  source: JmodTask.java:585
      objects: 0x10a70e630 0x10a70e6e8 0x12a3da7b0 0x12a3da828 0x12a3da850
     bp: 0x000000014a8160c8  method: void jdk/tools/jmod/JmodTask$JmodFileWriter.write(jdk.tools.jmod.JmodOutputStream)  source: JmodTask.java:481
      objects: 0x10a70e630 0x10a70e6e8
     bp: 0x000000014a816118  method: boolean jdk/tools/jmod/JmodTask.create()  source: JmodTask.java:438
      objects: 0x10a70e700 0x10a70e738 0x10a70e6e8
     bp: 0x000000014a816158  method: int jdk/tools/jmod/JmodTask.run(String[])  source: JmodTask.java:208
      objects: 0x10a70e770
     bp: 0x000000014a816188  method: void jdk/tools/jmod/Main.main(String[])  source: Main.java:34
      objects: <no objects in this frame>

The javacore has the following thread which is very similar to the core dump which fails in loadCrypto.

1XMTHDINFO     Thread Details
NULL
3XMTHREADINFO      "main" J9VMThread:0x000000012B00AF00, omrthread_t:0x000000015C00B650, java/lang/Thread:0x000000010A5FDD88, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000010A602268)
3XMTHREADINFO1            (native thread ID:0x243348A, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
3XMTHREADINFO2            (native stack address range from:0x000000016B49C000, to:0x000000016B4DF000, size:0x43000)
3XMCPUTIME               CPU usage total: 0.705739000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=1301264 (0x13DB10)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.loadCrypto(Native Method)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.loadCryptoLibraries(NativeCrypto.java:79)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.lambda$new$0(NativeCrypto.java:97)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto$$Lambda$96/0x0000000000000000.run(Bytecode PC:0)
4XESTACKTRACE                at java/security/AccessController.doPrivileged(AccessController.java:690)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.<init>(NativeCrypto.java:97)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto$InstanceHolder.<clinit>(NativeCrypto.java:64)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.getVersionIfAvailable(NativeCrypto.java:119)
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.isAllowedAndLoaded(NativeCrypto.java:108)
4XESTACKTRACE                at sun/security/provider/SunEntries.<init>(SunEntries.java:226)
4XESTACKTRACE                at sun/security/provider/Sun.<init>(Sun.java:54)
5XESTACKTRACE                   (entered lock: sun/security/jca/ProviderConfig@0x0000000129EC72D0, entry count: 1)
4XESTACKTRACE                at sun/security/jca/ProviderConfig.getProvider(ProviderConfig.java:189)
4XESTACKTRACE                at sun/security/jca/ProviderList.getProvider(ProviderList.java:290)
4XESTACKTRACE                at sun/security/jca/ProviderList.getService(ProviderList.java:404)
4XESTACKTRACE                at sun/security/jca/GetInstance.getInstance(GetInstance.java:157)
4XESTACKTRACE                at java/security/Security.getImpl(Security.java:716)
4XESTACKTRACE                at java/security/MessageDigest.getInstance(MessageDigest.java:178)
4XESTACKTRACE                at jdk/internal/module/ModuleHashes.computeHash(ModuleHashes.java:113)
4XESTACKTRACE                at jdk/internal/module/ModuleHashes.generate(ModuleHashes.java:167)
4XESTACKTRACE                at jdk/internal/module/ModuleHashesBuilder.lambda$computeHashes$3(ModuleHashesBuilder.java:120)
4XESTACKTRACE                at jdk/internal/module/ModuleHashesBuilder$$Lambda$89/0x0000000000000000.accept(Bytecode PC:20)
4XESTACKTRACE                at java/util/stream/ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183(Compiled Code))
4XESTACKTRACE                at java/util/stream/ReferencePipeline$2$1.accept(ReferencePipeline.java:177(Compiled Code))
4XESTACKTRACE                at java/util/ArrayDeque$DeqSpliterator.forEachRemaining(ArrayDeque.java:849)
4XESTACKTRACE                at java/util/stream/AbstractPipeline.copyInto(AbstractPipeline.java:497(Compiled Code))
4XESTACKTRACE                at java/util/stream/AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:487)
4XESTACKTRACE                at java/util/stream/ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
4XESTACKTRACE                at java/util/stream/ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
4XESTACKTRACE                at java/util/stream/AbstractPipeline.evaluate(AbstractPipeline.java:239(Compiled Code))
4XESTACKTRACE                at java/util/stream/ReferencePipeline.forEach(ReferencePipeline.java:497)
4XESTACKTRACE                at jdk/internal/module/ModuleHashesBuilder.computeHashes(ModuleHashesBuilder.java:104)
4XESTACKTRACE                at jdk/tools/jmod/JmodTask$Hasher.computeHashes(JmodTask.java:926)
4XESTACKTRACE                at jdk/tools/jmod/JmodTask$JmodFileWriter.computeHashes(JmodTask.java:654)
4XESTACKTRACE                at jdk/tools/jmod/JmodTask$JmodFileWriter.writeModuleInfo(JmodTask.java:585)
4XESTACKTRACE                at jdk/tools/jmod/JmodTask$JmodFileWriter.write(JmodTask.java:481)
4XESTACKTRACE                at jdk/tools/jmod/JmodTask.create(JmodTask.java:438)
4XESTACKTRACE                at jdk/tools/jmod/JmodTask.run(JmodTask.java:208)
4XESTACKTRACE                at jdk/tools/jmod/Main.main(Main.java:34)
3XMTHREADINFO3           No native callstack available for this thread

This does seem like it is related to loading of OpenSSL 3, or the JNI code related to calling OpenSSL. I did not however see a native trace here to see where the error occurs.

Is the only known difference between a working build and this failing build the openssl 3 bundling upgrade? I can see that the docker image was also updated in PR #14900 which I assume contains additional build dependencies such as perl packages?

pshipton commented 1 year ago

Is the only known difference between a working build and this failing build the openssl 3 bundling upgrade?

Pretty much. The nightly builds have always worked fine, and the only change here is the OpenSSL version. The docker image update was only for alinux, and it should have only added perl-IPC-Cmd which you can't compile OpenSSL v3 without.

I did not however see a native trace here to see where the error occurs.

You need to use the native debugger (lldb), probably the same machine where the crash occurred, or maybe a machine which is the same OS version would work.

pshipton commented 1 year ago

You need to use the native debugger (lldb), probably the same machine where the crash occurred, or maybe a machine which is the same OS version would work.

It might be easier to build on your machine and see if the crash is recreated.

jasonkatonica commented 1 year ago

it should have only added perl-IPC-Cmd which you can't compile OpenSSL v3 without

Right previous bundled builds would hit this perl dependency problem so we must be past that issue with the perl-IPC-Cmd added.

Our code for loading the dll on mac currently loads the the DLL names in a specific order with specific names. So i am not sure this code will work unless the general name libcrypto.dylib is available / built on the system. I did check the diagnostics file and I am able to only see a libcrypto.3.dylib file not a libcrypto.dylib file in the Build_JDK11_aarch64_mac_Personal-342-20230601-074019-diagnostics/build/macosx-aarch64-normal-server-release/jdk/lib directory. Notice that we do not explicitly load the libcrypto.3*.dylibdll name in the code. In theory I would not expect this to cause a crash, even if the library is not found, but its something for us to keep in mind and try.

I will check into trying to build openjdk on my local mac. If the build fails I can try to use lldb to read the native trace.

KostasTsiounis commented 1 year ago

According to several posted issues about the unsafe loading of the libcrypto library (such as this), loading the fallback library without a version (i.e., libcrypto.dylib or /usr/lib/libcrypto.dylib) produces a fatal warning that results in an abort.

KostasTsiounis commented 1 year ago

@jasonkatonica wrote a C program to reproduce that:

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

int main() {
    void *libssl_handle;

    // Load the OpenSSL library dynamically using the name libcrypto.3.dylib
    libssl_handle = dlopen("libcrypto.3.dylib", RTLD_LAZY);
    if (!libssl_handle) {
        fprintf(stderr, "Failed to load OpenSSL library with name libcrypto.3.dylib: %s\n", dlerror());
    } else {
        printf("OpenSSL library named libcrypto.3.dylib loaded successfully!\n");
        dlclose(libssl_handle);
        libssl_handle = NULL;
    }

    // Load the OpenSSL library dynamically using the name libcrypto.3.dylib
    libssl_handle = dlopen("libcrypto.dylib", RTLD_LAZY);
    if (!libssl_handle) {
        fprintf(stderr, "Failed to load OpenSSL library with name libcrypto.dylib: %s\n", dlerror());
    } else {
        printf("OpenSSL library named libcrypto.dylib loaded successfully!\n");
        dlclose(libssl_handle);
        libssl_handle = NULL;
    }

    return 0;
}
KostasTsiounis commented 1 year ago

This produces the following message:

OpenSSL library named libcrypto.3.dylib loaded successfully!
WARNING: /Users/kostastsiounis/ossl_tests/version_test is loading libcrypto in an unsafe way
zsh: abort      ./version_test
jasonkatonica commented 1 year ago

A fix for this issue has been merged into all Java releases except for Java 8.

Java next fix for AIX: ibmruntimes/openj9-openjdk-jdk#632

Java next fix for other platforms: ibmruntimes/openj9-openjdk-jdk#616

Multi platforms OpenJDK21: ibmruntimes/openj9-openjdk-jdk21#21

Multi platforms OpenJDK17: ibmruntimes/openj9-openjdk-jdk17#240

Multi platforms OpenJDK11: ibmruntimes/openj9-openjdk-jdk11#685

We will need to backport a fix for Java 8 also.

pshipton commented 1 year ago

https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/683 is merged for jdk8.