search

alibaba / arthas

Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas
https://arthas.aliyun.com/
Apache License 2.0
35.7k stars 7.51k forks source link

thread 指令导致 jvm crash #2931

Closed carl10086 closed 2 weeks ago

carl10086 commented 3 weeks ago

环境信息

重现问题的步骤

attach 执行 thread 指令

期望的结果

正常运行

实际运行的结果

jvm crash

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007efc4356d1d9, pid=271417, tid=1789212
#
# JRE version: OpenJDK Runtime Environment (21.0.2+13) (build 21.0.2+13-58)
# Java VM: OpenJDK 64-Bit Server VM (21.0.2+13-58, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x961d9]  pthread_getcpuclockid+0x9
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/data/app/voice-web-default/dist_1730257057/core.271417)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -javaagent:/home/aitogether/voice-web-default/apache-skywalking-java-agent/skywalking-agent.jar -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dspring.profiles.active=test -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DJM.LOG.PATH=/home/aitogether/voice-web-default/logs/ -DJM.SNAPSHOT.PATH=/home/aitogether/voice-web-default -Xmx1024m -Xms1024m -Xss256k -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/aitogether/voice-web-default/logs/java.hprof -Xlog:gc*,gc+age=trace,safepoint:file=/home/aitogether/voice-web-default/logs/gc.log:utctime,pid,tags:filecount=7,filesize=2m /opt/data/app/voice-web-default/dist_1730257057/voice-web-1.0.0-SNAPSHOT.jar

Host: Intel(R) Xeon(R) Platinum 8457C, 2 cores, 3G, Ubuntu 22.04.1 LTS
Time: Thu Oct 31 17:02:48 2024 CST elapsed time: 91402.773632 seconds (1d 1h 23m 22s)

---------------  T H R E A D  ---------------

Current thread (0x00007efc20dacbe0):  JavaThread "arthas-command-execute" daemon [_thread_in_vm, id=1789212, stack(0x00007efb8567a000,0x00007efb856ba000) (256K)] _threads_hazard_ptr=0x00007efc20dad380

Stack: [0x00007efb8567a000,0x00007efb856ba000],  sp=0x00007efb856b8058,  free space=248k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x961d9]  pthread_getcpuclockid+0x9
V  [libjvm.so+0xbff0ec]  jmm_GetThreadCpuTimeWithKind+0x19c
j  sun.management.ThreadImpl.getThreadTotalCpuTime0(J)J+0 java.management@21.0.2
j  sun.management.ThreadImpl.getThreadCpuTime([J)[J+82 java.management@21.0.2
j  com.sun.management.internal.HotSpotThreadImpl.getThreadCpuTime([J)[J+2 jdk.management@21.0.2
j  sun.management.ThreadImpl.getThreadCpuTime(J)J+10 java.management@21.0.2
j  com.taobao.arthas.core.command.monitor200.ThreadSampler.sample(Ljava/util/Collection;)Ljava/util/List;+73
j  com.taobao.arthas.core.command.monitor200.ThreadCommand.processAllThreads(Lcom/taobao/arthas/core/shell/command/CommandProcess;)Lcom/taobao/arthas/core/shell/command/ExitStatus;+302
j  com.taobao.arthas.core.command.monitor200.ThreadCommand.process(Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+52
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+34
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(Lcom/taobao/arthas/core/shell/command/impl/AnnotatedCommandImpl;Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+2
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+5
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(Ljava/lang/Object;)V+5
j  com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run()V+11
J 16212 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; java.base@21.0.2 (14 bytes) @ 0x00007efc2c886f74 [0x00007efc2c886f20+0x0000000000000054]
J 25469 c1 java.util.concurrent.FutureTask.run()V java.base@21.0.2 (123 bytes) @ 0x00007efc2683794c [0x00007efc26837220+0x000000000000072c]
J 25235 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V java.base@21.0.2 (57 bytes) @ 0x00007efc2c939010 [0x00007efc2c938e00+0x0000000000000210]
J 26525 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V java.base@21.0.2 (187 bytes) @ 0x00007efc2cd62adc [0x00007efc2cd62860+0x000000000000027c]
J 29905 c2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V java.base@21.0.2 (9 bytes) @ 0x00007efc2d3fa770 [0x00007efc2d3fa720+0x0000000000000050]
J 29839 c2 java.lang.Thread.run()V java.base@21.0.2 (23 bytes) @ 0x00007efc2d1a5aa4 [0x00007efc2d1a5a20+0x0000000000000084]
v  ~StubRoutines::call_stub 0x00007efc2c069cc6
V  [libjvm.so+0x8e28d5]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x2e5
V  [libjvm.so+0x8e4252]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1d2
V  [libjvm.so+0x9b20de]  thread_entry(JavaThread*, JavaThread*)+0x8e
V  [libjvm.so+0x8f8a78]  JavaThread::thread_main_inner() [clone .part.0]+0xb8
V  [libjvm.so+0xe9e998]  Thread::call_run()+0xa8
V  [libjvm.so+0xcc444a]  thread_native_entry(Thread*)+0xda
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sun.management.ThreadImpl.getThreadTotalCpuTime0(J)J+0 java.management@21.0.2
j  sun.management.ThreadImpl.getThreadCpuTime([J)[J+82 java.management@21.0.2
j  com.sun.management.internal.HotSpotThreadImpl.getThreadCpuTime([J)[J+2 jdk.management@21.0.2
j  sun.management.ThreadImpl.getThreadCpuTime(J)J+10 java.management@21.0.2
j  com.taobao.arthas.core.command.monitor200.ThreadSampler.sample(Ljava/util/Collection;)Ljava/util/List;+73
j  com.taobao.arthas.core.command.monitor200.ThreadCommand.processAllThreads(Lcom/taobao/arthas/core/shell/command/CommandProcess;)Lcom/taobao/arthas/core/shell/command/ExitStatus;+302
j  com.taobao.arthas.core.command.monitor200.ThreadCommand.process(Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+52
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+34
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(Lcom/taobao/arthas/core/shell/command/impl/AnnotatedCommandImpl;Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+2
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(Lcom/taobao/arthas/core/shell/command/CommandProcess;)V+5
j  com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(Ljava/lang/Object;)V+5
j  com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run()V+11
J 16212 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; java.base@21.0.2 (14 bytes) @ 0x00007efc2c886f74 [0x00007efc2c886f20+0x0000000000000054]
J 25469 c1 java.util.concurrent.FutureTask.run()V java.base@21.0.2 (123 bytes) @ 0x00007efc2683794c [0x00007efc26837220+0x000000000000072c]
J 25235 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V java.base@21.0.2 (57 bytes) @ 0x00007efc2c939010 [0x00007efc2c938e00+0x0000000000000210]
J 26525 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V java.base@21.0.2 (187 bytes) @ 0x00007efc2cd62adc [0x00007efc2cd62860+0x000000000000027c]
J 29905 c2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V java.base@21.0.2 (9 bytes) @ 0x00007efc2d3fa770 [0x00007efc2d3fa720+0x0000000000000050]
J 29839 c2 java.lang.Thread.run()V java.base@21.0.2 (23 bytes) @ 0x00007efc2d1a5aa4 [0x00007efc2d1a5a20+0x0000000000000084]
v  ~StubRoutines::call_stub 0x00007efc2c069cc6

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007efae5d9d910

Registers:
RAX=0x00007efc4356d1d0, RBX=0x00007efc20dacbe0, RCX=0x0000000000000000, RDX=0x0000000000000048
RSP=0x00007efb856b8058, RBP=0x00007efb856b8080, RSI=0x00007efb856b806c, RDI=0x00007efae5d9d640
R8 =0x0000000000000000, R9 =0x00000000000000f7, R10=0x00007efc2c075a51, R11=0x00000000c34fbe28
R12=0x00007efc0c8988c0, R13=0x0000000000000001, R14=0x00007efb856b8090, R15=0x00007efc20dacbe0
RIP=0x00007efc4356d1d9, EFLAGS=0x0000000000010202, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Register to memory mapping:

RAX=0x00007efc4356d1d0: pthread_getcpuclockid+0x0000000000000000 in /lib/x86_64-linux-gnu/libc.so.6 at 0x00007efc434d7000
RBX=0x00007efc20dacbe0 is a thread
RCX=0x0 is null
RDX=0x0000000000000048 is an unknown value
RSP=0x00007efb856b8058 is pointing into the stack for thread: 0x00007efc20dacbe0
RBP=0x00007efb856b8080 is pointing into the stack for thread: 0x00007efc20dacbe0
RSI=0x00007efb856b806c is pointing into the stack for thread: 0x00007efc20dacbe0
RDI=0x00007efae5d9d640 is pointing into the stack for thread: 0x00007efc0c8988c0
R8 =0x0 is null
R9 =0x00000000000000f7 is an unknown value
R10=0x00007efc2c075a51 is at code_begin+1009 in an Interpreter codelet
native method entry point (kind = native)  [0x00007efc2c075660, 0x00007efc2c0761d0]  2928 bytes
R11=0x00000000c34fbe28 is an oop: java.lang.Class
{0x00000000c34fbe28} - klass: 'java/lang/Class'
 - ---- fields (total size 14 words):
 - private volatile transient 'classRedefinedCount' 'I' @12  0 (0x00000000)
 - injected 'klass' 'J' @16  139619859155168 (0x00007efbc81bc8e0)
 - injected 'array_klass' 'J' @24  0 (0x0000000000000000)
 - injected 'oop_size' 'I' @32  14 (0x0000000e)
 - injected 'static_oop_field_count' 'I' @36  0 (0x00000000)
 - private volatile transient 'cachedConstructor' 'Ljava/lang/reflect/Constructor;' @40  null (0x00000000)
 - private transient 'name' 'Ljava/lang/String;' @44  "sun.management.ThreadImpl"{0x00000000c34fc7d8} (0xc34fc7d8)
 - private transient 'module' 'Ljava/lang/Module;' @48  a 'java/lang/Module'{0x00000000c028d9a0} (0xc028d9a0)
 - private final 'classLoader' 'Ljava/lang/ClassLoader;' @52  null (0x00000000)
 - private transient 'classData' 'Ljava/lang/Object;' @56  null (0x00000000)
 - private transient 'packageName' 'Ljava/lang/String;' @60  "sun.management"{0x00000000ffe6bae8} (0xffe6bae8)
 - private final 'componentType' 'Ljava/lang/Class;' @64  null (0x00000000)
 - private volatile transient 'reflectionData' 'Ljava/lang/ref/SoftReference;' @68  a 'java/lang/ref/SoftReference'{0x00000000c45b65b0} (0xc45b65b0)
 - private volatile transient 'genericInfo' 'Lsun/reflect/generics/repository/ClassRepository;' @72  null (0x00000000)
 - private volatile transient 'enumConstants' '[Ljava/lang/Object;' @76  null (0x00000000)
 - private volatile transient 'enumConstantDirectory' 'Ljava/util/Map;' @80  null (0x00000000)
 - private volatile transient 'annotationData' 'Ljava/lang/Class$AnnotationData;' @84  null (0x00000000)
 - private volatile transient 'annotationType' 'Lsun/reflect/annotation/AnnotationType;' @88  null (0x00000000)
 - transient 'classValueMap' 'Ljava/lang/ClassValue$ClassValueMap;' @92  null (0x00000000)
 - injected 'protection_domain' 'Ljava/lang/Object;' @96  null (0x00000000)
 - injected 'signers_name' 'Ljava/lang/Object;' @100  null (0x00000000)
 - injected 'source_file' 'Ljava/lang/Object;' @104  null (0x00000000)
 - signature: Lsun/management/ThreadImpl;
 - ---- static fields (0):
R12=0x00007efc0c8988c0 is a thread
R13=0x0000000000000001 is an unknown value
R14=0x00007efb856b8090 is pointing into the stack for thread: 0x00007efc20dacbe0
R15=0x00007efc20dacbe0 is a thread
hengyunabc commented 3 weeks ago

这个看起来是 jvm 本身的问题，是调用了 jvm 自身的 api：

https://github.com/alibaba/arthas/blob/8200afb3cca1fbcdc97d5ebc233b372e1469b76d/core/src/main/java/com/taobao/arthas/core/command/monitor200/ThreadSampler.java#L43

carl10086 commented 3 weeks ago

这个看起来是 jvm 本身的问题，是调用了 jvm 自身的 api：

https://github.com/alibaba/arthas/blob/8200afb3cca1fbcdc97d5ebc233b372e1469b76d/core/src/main/java/com/taobao/arthas/core/command/monitor200/ThreadSampler.java#L43

好的，我换了 azuul 的 jdk 再测试一下