search

adoptium / adoptium-support

For end-user problems reported with our binary distributions
Apache License 2.0
45 stars 15 forks source link

JVM 21 crashes with SIGSEGV caused by virtual threads #1127

Open stepan-hrbacek-n-able opened 2 months ago

stepan-hrbacek-n-able commented 2 months ago

Please provide a brief summary of the bug

JVM crashes on SEGSEGV error (segmentation fault) with following error indicating that this is caused by java virtual threads:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9dab612127, pid=15662, tid=33827
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.1+12 (21.0.1+12) (build 21.0.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (21.0.1+12-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x69b127]  ContinuationEntry::set_enter_code(CompiledMethod*, int)+0x7
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -XX:+UseG1GC -XX:ParallelGCThreads=40 -XX:ConcGCThreads=40 -Xlog:gc*,safepoint:/var/log/n-central/jetty/gc.log:time,uptime,level,tags:filecount=5,filesize=100M -XX:+AlwaysPreTouch -XX:-ClassUnloading -XX:-OmitStackTraceInFastThrow --add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.time.zone=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED --add-opens=java.xml/jdk.xml.internal=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -XX:+UseNUMA -Xmx56054m -Xms16015m -XX:MetaspaceSize=1024m -Xss256K -Djava.awt.headless=true -Dlog4j2.configurationFile=file:////opt/nable/webapps/ROOT/WEB-INF/classes/log4j2.xml -Dlog4j2.formatMsgNoLookups=true -Dorg.apache.cxf.Logger=org.apache.cxf.common.logging.Slf4jLogger -Dorg.apache.commons.logging.LogFactory=org.apache.commons.logging.impl.LogFactoryImpl -Djava.security.egd=file:/dev/./urandom -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Daxis.ServerConfigFile=/com/nable/dmsapi/server-config.wsdd -Dnetworkaddress.cache.ttl=60 -Djdk.tls.ephemeralDHKeySize=2048 -Dognl.security.manager -Djetty.home=/opt/jetty -Djetty.base=/opt/nable -Djava.io.tmpdir=/opt/nable/tmp /opt/jetty/start.jar start-log-file=/var/log/n-central/jetty/jetty-start.log jetty.state=/opt/nable/jetty.state jetty-started.xml

Host: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz, 40 cores, 157G, CentOS Linux release 7.9.2009 (Core)
Time: Mon Jun 17 04:07:34 2024 CDT elapsed time: 73403.522546 seconds (0d 20h 23m 23s)

---------------  T H R E A D  ---------------

Current thread (0x00007f8ad4007980):  JavaThread "ForkJoinPool-1-worker-1" daemon [_thread_in_vm, id=33827, stack(0x00007f8a0cf9d000,0x00007f8a0cfdd000) (256K)]

Stack: [0x00007f8a0cf9d000,0x00007f8a0cfdd000],  sp=0x00007f8a0cfda7f8,  free space=245k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x69b127]  ContinuationEntry::set_enter_code(CompiledMethod*, int)+0x7
V  [libjvm.so+0xda4673]  AdapterHandlerLibrary::create_native_wrapper(methodHandle const&)+0x493
V  [libjvm.so+0xb75ec3]  LinkResolver::resolve_static_call(CallInfo&, LinkInfo const&, bool, JavaThread*)+0x2a3
V  [libjvm.so+0xb7666b]  LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle const&, int, Bytecodes::Code, JavaThread*)+0x1fb
V  [libjvm.so+0x8f7ee7]  InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x177
V  [libjvm.so+0x8f85a5]  InterpreterRuntime::resolve_from_cache(JavaThread*, Bytecodes::Code)+0x105
j  jdk.internal.vm.Continuation.run()V+122 java.base@21.0.1
j  java.lang.VirtualThread.runContinuation()V+71 java.base@21.0.1
j  java.lang.VirtualThread$$Lambda+0x00007f8e9dee3dd8.run()V+4 java.base@21.0.1
j  java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec()Z+4 java.base@21.0.1
j  java.util.concurrent.ForkJoinTask.doExec()I+10 java.base@21.0.1
j  java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Ljava/util/concurrent/ForkJoinTask;Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V+19 java.base@21.0.1
j  java.util.concurrent.ForkJoinPool.scan(Ljava/util/concurrent/ForkJoinPool$WorkQueue;II)I+211 java.base@21.0.1
j  java.util.concurrent.ForkJoinPool.runWorker(Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V+35 java.base@21.0.1
j  java.util.concurrent.ForkJoinWorkerThread.run()V+31 java.base@21.0.1

The only usage of virtual threads in the crashing program is using an executor created by:

java.util.concurrent.Executors.newVirtualThreadPerTaskExecutor()

The issue goes away after replacing it with a normal thread pool:

java.util.concurrent.Executors.newSingleThreadExecutor()

hs_err_pid15662.log

Did you test with the latest update version?

Please provide steps to reproduce where possible

No response

Expected Results

JVM does not crash when using virtual threads.

Actual Results

JVM crashes with SIGSEGV when using virtual threads.

What Java Version are you using?

OpenJDK Runtime Environment Temurin-21.0.1+12 (21.0.1+12) (build 21.0.1+12-LTS)

What is your operating system and platform?

Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz, 40 cores, 157G, CentOS x64 Linux release 7.9.2009 (Core)

How did you install Java?

RPM

Did it work before?

No response

Did you test with other Java versions?

No response

Relevant log output

No response

stepan-hrbacek-n-able commented 2 months ago

I was unfortunately not able to test this with the latest JVM 21 release because the issue only occurs in a customer production environment where changing JVM version is hard...

jerboaa commented 2 months ago

There have been several fixes in the loom area of latest JDK 21, so please be sure to update and see if it still reproduces with a later version (21.0.3 currently, or 21.0.4 in two weeks time). Unfortunately, we cannot fix 21.0.1 only. Fixes can only be delivered with latest update releases (if it still reproduces).

stepan-hrbacek-n-able commented 2 months ago

Thank you @jerboaa 👍