Crash on OpenJDK11 and Oracle JDK 8 on macOS

[greenhat]

Branch: develop

Steps to reproduce: run sbt test

Log for OpenJDK11:

[info] - applyModifier() - invalid block
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000001116703f0, pid=57166, tid=38147
#
# JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10)
# Java VM: OpenJDK 64-Bit Server VM (11.0.6+10, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libleveldbjni-64-1-5503110146537748033.8+0x373f0]  typeinfo for leveldb::DB+0x0
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/green/code/WORK/ergo/hs_err_pid57166.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/AdoptOpenJDK/openjdk-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Exception in thread "Thread-11" java.io.EOFException
    at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3117)
    at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1597)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:464)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.react(Framework.scala:818)
    at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1.run(Framework.scala:807)
    at java.base/java.lang.Thread.run(Thread.java:834)

Log for OracleJDK8:

[info] - applyModifier() - invalid block
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000001303c33f0, pid=65014, tid=0x0000000000005903
#
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libleveldbjni-64-1-5184192378358824251.8+0x373f0]  _ZTIN7leveldb2DBE+0x0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/green/code/WORK/ergo/hs_err_pid65014.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Exception in thread "Thread-10" java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2960)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1540)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.react(Framework.scala:818)
    at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1.run(Framework.scala:807)
    at java.lang.Thread.run(Thread.java:748)

[greenhat]

I've found logs with the same crashes when I ran 3.2.0 node on testnet. I've attached them. hs_err_pid2139.log hs_err_pid2452.log hs_err_pid3225.log hs_err_pid55029.log

[bjenkinsgit]

I had crashes on Mac OS X Catalina using OpenJDK 11. Installing ORACLE JDK 14 got me further, but ultimately crashed from leveldbjni.

[bjenkinsgit]

Still crashing under Mac OS X Catalina using ORACLE JDK 11. Runs for about 3 hours and then dies with a problem in leveldbjni. Here is the latest screen dump in the console when it dies:

16:39:33.247 WARN  [tor.default-dispatcher-62] s.core.network.SyncTracker - Trying to clear status for /94.130.108.41:9020, but it is not found
16:39:33.247 WARN  [tor.default-dispatcher-62] s.core.network.SyncTracker - Trying to clear last sync time for /94.130.108.41:9020, but it is not found
16:39:53.042 WARN  [tor.default-dispatcher-70] s.c.n.NetworkController - Connection to peer /78.46.93.239:9032 is already established
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x000000012b6213f0, pid=32699, tid=44551
#
# JRE version: Java(TM) SE Runtime Environment 18.9 (11.0.7+8) (build 11.0.7+8-LTS)
# Java VM: Java HotSpot(TM) 64-Bit Server VM 18.9 (11.0.7+8-LTS, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libleveldbjni-64-3-13370053891917898906.2+0x373f0]  typeinfo for leveldb::DB+0x0
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/bwj/Documents/SRC/ergosrc/hs_err_pid32699.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
./runErgoNode_mainnet.sh: line 1: 32699 Abort trap: 6           java -jar -Xmx3G -Dlogback.stdout.level=WARN ergo-3.2.2.jar --mainnet -c ./ergo_mainnet.conf

[1]  + exit 134   ./runErgoNode_mainnet.sh

[kushti]

I think the only solution we can do (without switching to other DB than LevelDB, which is probably an overkill as LevelDB works pretty well for now) is to switch automatically to pure-Java LevelDB on MacOS (like we're doing for platforms for which there are no LevelDB binaries, such as 32-bit ARMs)