search

sofastack / sofa-jraft

A production-grade java implementation of RAFT consensus algorithm.
https://www.sofastack.tech/projects/sofa-jraft/
Apache License 2.0
3.52k stars 1.12k forks source link

Unrecycled RPC threads cause OOM #1104

Closed Brokenice0415 closed 1 month ago

Brokenice0415 commented 1 month ago

Describe the bug

I started and stopped jraft server multiple times via ProcessBuilder and triggered the OOM.

My guess is that the cause of OOM is that I didn't call shutdown but killed the process when stopping it, and RPC-related resources weren't properly recycled.

Actual behavior

#  Out of Memory Error (os_linux.cpp:2787), pid=3492334, tid=3689298
#
# JRE version: OpenJDK Runtime Environment Zulu17.38+21-CA (17.0.5+8) (build 17.0.5+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.38+21-CA (17.0.5+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, linux-amd64)
#

---------------  S U M M A R Y ------------

Command Line: -Xmx1800m -XX:-OmitStackTraceInFastThrow -XX:+UseParallelGC -XX:+IgnoreUnrecognizedVMOptions -XX:+CriticalJNINatives -XX:+EnableDynamicAgentLoading --add-opens=java.base/java.lang=ALL-UNNAMED 

Host: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 16 cores, 62G, Ubuntu 22.04.3 LTS
Time: Fri May 10 02:29:21 2024 CST elapsed time: 9493.794026 seconds (0d 2h 38m 13s)

---------------  T H R E A D  ---------------

Current thread (0x00007ff6e4981c20):  JavaThread "JRaft-RPC-Processor-0" daemon [_thread_new, id=3689298, stack(0x00007feedbd00000,0x00007feedbe00000)]

Stack: [0x00007feedbd00000,0x00007feedbe00000],  sp=0x00007feedbdfe8f0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xeffe02]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1a2
V  [libjvm.so+0xf00aeb]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V  [libjvm.so+0x601082]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xc2
V  [libjvm.so+0xbcbf48]  os::pd_commit_memory(char*, unsigned long, bool)+0xd8
V  [libjvm.so+0xbc57ef]  os::commit_memory(char*, unsigned long, bool)+0x1f
V  [libjvm.so+0xbccd3e]  os::pd_create_stack_guard_pages(char*, unsigned long)+0x6e
V  [libjvm.so+0xdbc588]  StackOverflow::create_stack_guard_pages()+0x58
V  [libjvm.so+0xe83a23]  JavaThread::run()+0x23
V  [libjvm.so+0xe8680d]  Thread::call_run()+0x6d
V  [libjvm.so+0xbcece1]  thread_native_entry(Thread*)+0xe1

Java Threads: ( => current thread )
  0x0000560233e93d80 JavaThread "main" [_thread_blocked, id=3492334, stack(0x00007ffe066a1000,0x00007ffe067a1000)]
  0x0000560233ff84b0 JavaThread "Reference Handler" daemon [_thread_blocked, id=3492338, stack(0x00007ff7db92e000,0x00007ff7dba2e000)]
  0x0000560233ff98e0 JavaThread "Finalizer" daemon [_thread_blocked, id=3492339, stack(0x00007ff7db82e000,0x00007ff7db92e000)]
  0x0000560233ffff70 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3492340, stack(0x00007ff7daeb6000,0x00007ff7dafb6000)]
  0x0000560234001310 JavaThread "Service Thread" daemon [_thread_blocked, id=3492341, stack(0x00007ff7dadb6000,0x00007ff7daeb6000)]
  0x0000560234002710 JavaThread "Monitor Deflation Thread" daemon [_thread_blocked, id=3492342, stack(0x00007ff7dacb6000,0x00007ff7dadb6000)]
  0x00005602340040b0 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3492343, stack(0x00007ff7dabb6000,0x00007ff7dacb6000)]
  0x00005602340055d0 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=3492344, stack(0x00007ff7daab6000,0x00007ff7dabb6000)]
  0x000056023400ea20 JavaThread "Sweeper thread" daemon [_thread_blocked, id=3492345, stack(0x00007ff7da9b6000,0x00007ff7daab6000)]
  0x0000560234054e60 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=3492346, stack(0x00007ff7da8b6000,0x00007ff7da9b6000)]
  0x0000560234057170 JavaThread "Notification Thread" daemon [_thread_blocked, id=3492347, stack(0x00007ff7da7b6000,0x00007ff7da8b6000)]
  0x00005602340638b0 JavaThread "process reaper" daemon [_thread_in_native, id=3492352, stack(0x00007ff7dc46d000,0x00007ff7dc48f000)]
  0x00007ff7c4000ff0 JavaThread "Attach Listener" daemon [_thread_blocked, id=3492370, stack(0x00007ff7da4b4000,0x00007ff7da5b4000)]
  0x0000560234ec8020 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=3492694, stack(0x00007ff7d98b4000,0x00007ff7d99b4000)]
  0x0000560234ecb870 JavaThread "RMI Reaper" [_thread_blocked, id=3492695, stack(0x00007ff7d9ab4000,0x00007ff7d9bb4000)]
  0x0000560234ecd0f0 JavaThread "RMI GC Daemon" daemon [_thread_blocked, id=3492696, stack(0x00007ff7d9eb4000,0x00007ff7d9fb4000)]
  0x0000560234ec2da0 JavaThread "RMI TCP Accept-2599" daemon [_thread_in_native, id=3492697, stack(0x00007ff7d9db4000,0x00007ff7d9eb4000)]
  0x0000560234f51150 JavaThread "process reaper" daemon [_thread_in_native, id=3492701, stack(0x00007ff7d9703000,0x00007ff7d9725000)]
  0x0000560234f52a70 JavaThread "process reaper" daemon [_thread_in_native, id=3492704, stack(0x00007ff7d96e1000,0x00007ff7d9703000)]
  0x00007ff78c00f4f0 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=3492782, stack(0x00007ff7d94ee000,0x00007ff7d95ee000)]
  0x00007ff7187dfd90 JavaThread "RpcTaskScannerThread-8-thread-1" daemon [_thread_blocked, id=3493345, stack(0x00007ff740300000,0x00007ff740400000)]
  0x00007ff704005690 JavaThread "RpcTaskScannerThread-9-thread-1" daemon [_thread_blocked, id=3493346, stack(0x00007ff740200000,0x00007ff740300000)]
  0x00007ff7040420b0 JavaThread "bolt-netty-client-worker-1-thread-1" daemon [_thread_in_native, id=3493349, stack(0x00007ff7410fe000,0x00007ff7411fe000)]
  0x00007ff71859d120 JavaThread "bolt-netty-client-worker-1-thread-2" daemon [_thread_in_native, id=3493350, stack(0x00007ff740100000,0x00007ff740200000)]
  0x00007ff710040d60 JavaThread "Bolt-conn-event-executor-5-thread-1" daemon [_thread_blocked, id=3493353, stack(0x00007ff740000000,0x00007ff740100000)]
  0x00007ff70000d3b0 JavaThread "Bolt-conn-event-executor-4-thread-1" daemon [_thread_blocked, id=3493354, stack(0x00007ff6fbf00000,0x00007ff6fc000000)]
  0x00007ff710138e80 JavaThread "Bolt-default-executor-6-thread-1" daemon [_thread_blocked, id=3493358, stack(0x00007ff6f9dfe000,0x00007ff6f9efe000)]
  0x00007ff70000ff90 JavaThread "Bolt-default-executor-6-thread-2" daemon [_thread_blocked, id=3493359, stack(0x00007ff6f9cfe000,0x00007ff6f9dfe000)]
  0x00007ff70406d6b0 JavaThread "DefaultTimer10-10-thread-1" daemon [_thread_blocked, id=3493360, stack(0x00007ff6f9bfe000,0x00007ff6f9cfe000)]
  0x00007ff700012880 JavaThread "Bolt-default-executor-6-thread-3" daemon [_thread_blocked, id=3493363, stack(0x00007ff6f9afe000,0x00007ff6f9bfe000)]
  0x00007ff710139f00 JavaThread "Bolt-default-executor-6-thread-4" daemon [_thread_blocked, id=3493364, stack(0x00007ff6f99fe000,0x00007ff6f9afe000)]
  0x00007ff6d80012f0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493365, stack(0x00007ff6f98fe000,0x00007ff6f99fe000)]
  0x00007ff6e4003390 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493366, stack(0x00007ff6f97fe000,0x00007ff6f98fe000)]
  0x00007ff6dc004d90 JavaThread "RpcTaskScannerThread-13-thread-1" daemon [_thread_blocked, id=3493368, stack(0x00007ff6f95fe000,0x00007ff6f96fe000)]
  0x00007ff6dc014750 JavaThread "bolt-netty-client-worker-1-thread-3" daemon [_thread_in_native, id=3493369, stack(0x00007ff6f94fe000,0x00007ff6f95fe000)]
  0x00007ff6e4001ab0 JavaThread "Bolt-conn-event-executor-12-thread-1" daemon [_thread_blocked, id=3493371, stack(0x00007ff6f93fe000,0x00007ff6f94fe000)]
  0x00007ff6e4005fd0 JavaThread "Bolt-default-executor-6-thread-5" daemon [_thread_blocked, id=3493372, stack(0x00007ff6f82fd000,0x00007ff6f83fd000)]
  0x00007ff6e4007170 JavaThread "Bolt-default-executor-6-thread-6" daemon [_thread_blocked, id=3493374, stack(0x00007ff6f81fd000,0x00007ff6f82fd000)]
  0x00007ff6e8001700 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493375, stack(0x00007ff6f80fd000,0x00007ff6f81fd000)]
  0x00007ff700014560 JavaThread "RpcTaskScannerThread-16-thread-1" daemon [_thread_blocked, id=3493379, stack(0x00007ff6d7d00000,0x00007ff6d7e00000)]
  0x00007ff700015b00 JavaThread "bolt-netty-client-worker-1-thread-4" daemon [_thread_in_native, id=3493380, stack(0x00007ff6d7c00000,0x00007ff6d7d00000)]
  0x00007ff708001720 JavaThread "Bolt-conn-event-executor-15-thread-1" daemon [_thread_blocked, id=3493382, stack(0x00007ff6d7b00000,0x00007ff6d7c00000)]
  0x00007ff708002a50 JavaThread "Bolt-default-executor-6-thread-7" daemon [_thread_in_Java, id=3493383, stack(0x00007ff6d69ff000,0x00007ff6d6aff000)]
  0x00007ff708003c90 JavaThread "Bolt-default-executor-6-thread-8" daemon [_thread_blocked, id=3493385, stack(0x00007ff6d68ff000,0x00007ff6d69ff000)]
  0x00007ff70c003710 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493386, stack(0x00007ff6d67ff000,0x00007ff6d68ff000)]
  0x00007ff7000187e0 JavaThread "Bolt-default-executor-6-thread-9" daemon [_thread_blocked, id=3493388, stack(0x00007ff6d66ff000,0x00007ff6d67ff000)]
  0x00007ff6e4008360 JavaThread "Bolt-default-executor-6-thread-10" daemon [_thread_blocked, id=3493389, stack(0x00007ff6d65ff000,0x00007ff6d66ff000)]
  0x00007ff7080052b0 JavaThread "Bolt-default-executor-6-thread-11" daemon [_thread_blocked, id=3493392, stack(0x00007ff6d64ff000,0x00007ff6d65ff000)]
  0x00007ff710150720 JavaThread "Bolt-default-executor-6-thread-12" daemon [_thread_blocked, id=3493393, stack(0x00007ff6d63ff000,0x00007ff6d64ff000)]
  0x00007ff738003790 JavaThread "RpcTaskScannerThread-19-thread-1" daemon [_thread_blocked, id=3493746, stack(0x00007ff6d5fff000,0x00007ff6d60ff000)]
  0x00007ff738004b70 JavaThread "bolt-netty-client-worker-1-thread-5" daemon [_thread_in_native, id=3493747, stack(0x00007ff6d5eff000,0x00007ff6d5fff000)]
  0x00007ff73c003a70 JavaThread "Bolt-conn-event-executor-18-thread-1" daemon [_thread_blocked, id=3493749, stack(0x00007ff6d5dff000,0x00007ff6d5eff000)]
  0x00007ff73c004e00 JavaThread "Bolt-default-executor-6-thread-13" daemon [_thread_blocked, id=3493753, stack(0x00007ff6d4cfe000,0x00007ff6d4dfe000)]
  0x00007ff73c006090 JavaThread "Bolt-default-executor-6-thread-14" daemon [_thread_blocked, id=3493755, stack(0x00007ff6d4bfe000,0x00007ff6d4cfe000)]
  0x00007ff750002df0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493756, stack(0x00007ff6d4afe000,0x00007ff6d4bfe000)]
  0x00007ff73c007320 JavaThread "Bolt-default-executor-6-thread-15" daemon [_thread_blocked, id=3493778, stack(0x00007ff6d46fe000,0x00007ff6d47fe000)]
  0x00007ff768251010 JavaThread "RpcTaskScannerThread-22-thread-1" daemon [_thread_blocked, id=3493781, stack(0x00007ff6d44fe000,0x00007ff6d45fe000)]
  0x00007ff768c1b7d0 JavaThread "bolt-netty-client-worker-1-thread-6" daemon [_thread_in_native, id=3493782, stack(0x00007ff6d43fe000,0x00007ff6d44fe000)]
  0x00007ff770005d00 JavaThread "Bolt-conn-event-executor-21-thread-1" daemon [_thread_blocked, id=3493784, stack(0x00007ff6d42fe000,0x00007ff6d43fe000)]
  0x00007ff770002240 JavaThread "Bolt-default-executor-6-thread-16" daemon [_thread_blocked, id=3493785, stack(0x00007ff6d31fd000,0x00007ff6d32fd000)]
  0x00007ff770003270 JavaThread "Bolt-default-executor-6-thread-17" daemon [_thread_blocked, id=3493787, stack(0x00007ff6d30fd000,0x00007ff6d31fd000)]
  0x00007ff774020180 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493788, stack(0x00007ff6d2ffd000,0x00007ff6d30fd000)]
  0x00007ff770006f40 JavaThread "Bolt-default-executor-6-thread-18" daemon [_thread_blocked, id=3493790, stack(0x00007ff6d2efd000,0x00007ff6d2ffd000)]
  0x00007ff788025c50 JavaThread "RpcTaskScannerThread-25-thread-1" daemon [_thread_blocked, id=3493794, stack(0x00007ff6d2cfd000,0x00007ff6d2dfd000)]
  0x00007ff78804d350 JavaThread "bolt-netty-client-worker-1-thread-7" daemon [_thread_in_native, id=3493795, stack(0x00007ff6d2bfd000,0x00007ff6d2cfd000)]
  0x00007ff79000aa60 JavaThread "Bolt-conn-event-executor-24-thread-1" daemon [_thread_blocked, id=3493797, stack(0x00007ff6d2afd000,0x00007ff6d2bfd000)]
  0x00007ff790007eb0 JavaThread "Bolt-default-executor-6-thread-19" daemon [_thread_blocked, id=3493798, stack(0x00007ff6d19fc000,0x00007ff6d1afc000)]
  0x00007ff790009280 JavaThread "Bolt-default-executor-6-thread-20" daemon [_thread_blocked, id=3493800, stack(0x00007ff6d18fc000,0x00007ff6d19fc000)]
  0x00007ff79400ff80 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493801, stack(0x00007ff6d17fc000,0x00007ff6d18fc000)]
  0x00007ff79c002090 JavaThread "RpcTaskScannerThread-28-thread-1" daemon [_thread_blocked, id=3493805, stack(0x00007ff6d15fc000,0x00007ff6d16fc000)]
  0x00007ff79c0038f0 JavaThread "bolt-netty-client-worker-1-thread-8" daemon [_thread_in_native, id=3493806, stack(0x00007ff6d14fc000,0x00007ff6d15fc000)]
  0x00007ff7a40015c0 JavaThread "Bolt-conn-event-executor-27-thread-1" daemon [_thread_blocked, id=3493808, stack(0x00007ff6d13fc000,0x00007ff6d14fc000)]
  0x00007ff6d8002ba0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493810, stack(0x00007ff6d02fb000,0x00007ff6d03fb000)]
  0x00007ff7b80015c0 JavaThread "RpcTaskScannerThread-31-thread-1" daemon [_thread_blocked, id=3493814, stack(0x00007ff6d00fb000,0x00007ff6d01fb000)]
  0x00007ff7b8002eb0 JavaThread "bolt-netty-client-worker-1-thread-9" daemon [_thread_in_native, id=3493815, stack(0x00007ff6cfffb000,0x00007ff6d00fb000)]
  0x00007ff7c0ced620 JavaThread "Bolt-conn-event-executor-30-thread-1" daemon [_thread_blocked, id=3493817, stack(0x00007ff6cfefb000,0x00007ff6cfffb000)]
  0x00007ff6ec001720 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493819, stack(0x00007ff6cedfa000,0x00007ff6ceefa000)]
  0x00007ff7c4002720 JavaThread "RpcTaskScannerThread-34-thread-1" daemon [_thread_blocked, id=3493821, stack(0x00007ff6cebfa000,0x00007ff6cecfa000)]
  0x00007ff71013ecf0 JavaThread "Bolt-conn-event-executor-33-thread-1" daemon [_thread_blocked, id=3493823, stack(0x00007ff6ceafa000,0x00007ff6cebfa000)]
  0x00007ff71013f940 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493825, stack(0x00007ff6ce9fa000,0x00007ff6ceafa000)]
  0x00007ff7e0002fe0 JavaThread "RpcTaskScannerThread-37-thread-1" daemon [_thread_blocked, id=3493842, stack(0x00007ff6ce7fa000,0x00007ff6ce8fa000)]
  0x00007ff700019bd0 JavaThread "Bolt-conn-event-executor-36-thread-1" daemon [_thread_blocked, id=3493844, stack(0x00007ff6ce6fa000,0x00007ff6ce7fa000)]
  0x00007ff714003c70 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493846, stack(0x00007ff6ce5fa000,0x00007ff6ce6fa000)]
  0x00007ff6e4016330 JavaThread "RpcTaskScannerThread-40-thread-1" daemon [_thread_blocked, id=3493848, stack(0x00007ff6ce3fa000,0x00007ff6ce4fa000)]
  0x00007ff6e4021b40 JavaThread "Bolt-conn-event-executor-39-thread-1" daemon [_thread_blocked, id=3493850, stack(0x00007ff6ce2fa000,0x00007ff6ce3fa000)]
  0x00007ff71c0038b0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493852, stack(0x00007ff6ce1fa000,0x00007ff6ce2fa000)]
  0x00007ff6f0002e10 JavaThread "RpcTaskScannerThread-43-thread-1" daemon [_thread_blocked, id=3493860, stack(0x00007ff6cdffa000,0x00007ff6ce0fa000)]
  0x00007ff708006610 JavaThread "Bolt-conn-event-executor-42-thread-1" daemon [_thread_blocked, id=3493862, stack(0x00007ff6cdefa000,0x00007ff6cdffa000)]
  0x00007ff750004470 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493872, stack(0x00007ff6cddfa000,0x00007ff6cdefa000)]
  0x00007ff708008020 JavaThread "RpcTaskScannerThread-46-thread-1" daemon [_thread_blocked, id=3493876, stack(0x00007ff6cdbfa000,0x00007ff6cdcfa000)]
  0x00007ff790005100 JavaThread "Bolt-conn-event-executor-45-thread-1" daemon [_thread_blocked, id=3493878, stack(0x00007ff6cdafa000,0x00007ff6cdbfa000)]
  0x00007ff70c005450 JavaThread "RpcTaskScannerThread-49-thread-1" daemon [_thread_blocked, id=3493880, stack(0x00007ff6cd8fa000,0x00007ff6cd9fa000)]
  0x00007ff7a4002cc0 JavaThread "Bolt-conn-event-executor-48-thread-1" daemon [_thread_blocked, id=3493882, stack(0x00007ff6cd7fa000,0x00007ff6cd8fa000)]
  0x00007ff720003fa0 JavaThread "RpcTaskScannerThread-52-thread-1" daemon [_thread_blocked, id=3493884, stack(0x00007ff6cd5fa000,0x00007ff6cd6fa000)]
  0x00007ff7c0e96e90 JavaThread "Bolt-conn-event-executor-51-thread-1" daemon [_thread_blocked, id=3493886, stack(0x00007ff6cd4fa000,0x00007ff6cd5fa000)]
  0x00007ff794011240 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493889, stack(0x00007ff6cd3fa000,0x00007ff6cd4fa000)]
  0x00007ff7300060b0 JavaThread "RpcTaskScannerThread-55-thread-1" daemon [_thread_blocked, id=3493893, stack(0x00007ff6cd1fa000,0x00007ff6cd2fa000)]
  0x00007ff6e4022c70 JavaThread "Bolt-conn-event-executor-54-thread-1" daemon [_thread_blocked, id=3493895, stack(0x00007ff6cd0fa000,0x00007ff6cd1fa000)]
  0x00007ff73400cc70 JavaThread "RpcTaskScannerThread-58-thread-1" daemon [_thread_blocked, id=3493897, stack(0x00007ff6ccefa000,0x00007ff6ccffa000)]
  0x00007ff708009640 JavaThread "Bolt-conn-event-executor-57-thread-1" daemon [_thread_blocked, id=3493899, stack(0x00007ff6ccdfa000,0x00007ff6ccefa000)]
  0x00007ff744002d70 JavaThread "RpcTaskScannerThread-61-thread-1" daemon [_thread_blocked, id=3493901, stack(0x00007ff6ccbfa000,0x00007ff6cccfa000)]
  0x00007ff73c008cc0 JavaThread "Bolt-conn-event-executor-60-thread-1" daemon [_thread_blocked, id=3493902, stack(0x00007ff6ccafa000,0x00007ff6ccbfa000)]
  0x00007ff6e8003130 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493905, stack(0x00007ff6cc9fa000,0x00007ff6ccafa000)]
  0x00007ff714005b20 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493910, stack(0x00007ff6cc8fa000,0x00007ff6cc9fa000)]
  0x00007ff71c005780 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493917, stack(0x00007ff6cbf00000,0x00007ff6cc000000)]
  0x00007ff77c001890 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493921, stack(0x00007ff6cbe00000,0x00007ff6cbf00000)]
  0x00007ff6e80045d0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493926, stack(0x00007ff6cbd00000,0x00007ff6cbe00000)]
  0x00007ff6e4024100 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493931, stack(0x00007ff6cbc00000,0x00007ff6cbd00000)]
  0x00007ff6e80055d0 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493934, stack(0x00007ff6cbb00000,0x00007ff6cbc00000)]
  0x00007ff71013c380 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493935, stack(0x00007ff6cba00000,0x00007ff6cbb00000)]
  0x00007ff70c008780 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493937, stack(0x00007ff6cb900000,0x00007ff6cba00000)]
  0x00007ff714006fc0 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493938, stack(0x00007ff6cb800000,0x00007ff6cb900000)]
  0x00007ff720005960 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493939, stack(0x00007ff6cb700000,0x00007ff6cb800000)]
  0x00007ff71c006c30 JavaThread "JRaft-RPC-Processor-1" daemon [_thread_blocked, id=3493940, stack(0x00007ff6cb600000,0x00007ff6cb700000)]
  0x00007ff7440042d0 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493947, stack(0x00007ff6cb500000,0x00007ff6cb600000)]
  0x00007ff750005b90 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493948, stack(0x00007ff6cb400000,0x00007ff6cb500000)]
  0x00007ff778005fc0 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493949, stack(0x00007ff6cb300000,0x00007ff6cb400000)]
  0x00007ff728003f80 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493950, stack(0x00007ff6cb200000,0x00007ff6cb300000)]
  0x00007ff75c00bd80 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493951, stack(0x00007ff6cb100000,0x00007ff6cb200000)]
  0x00007ff77401db50 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493952, stack(0x00007ff6cb000000,0x00007ff6cb100000)]
  0x00007ff77c007df0 JavaThread "JRaft-RPC-Processor-2" daemon [_thread_blocked, id=3493953, stack(0x00007ff6caf00000,0x00007ff6cb000000)]
  0x00007ff79c005940 JavaThread "RpcTaskScannerThread-64-thread-1" daemon [_thread_blocked, id=3493968, stack(0x00007ff6cad00000,0x00007ff6cae00000)]
  0x00007ff790006700 JavaThread "Bolt-conn-event-executor-63-thread-1" daemon [_thread_blocked, id=3493969, stack(0x00007ff6cac00000,0x00007ff6cad00000)]
  0x00007ff6e80060f0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493970, stack(0x00007ff6cab00000,0x00007ff6cac00000)]
  0x00007ff7ac0020d0 JavaThread "RpcTaskScannerThread-67-thread-1" daemon [_thread_blocked, id=3493974, stack(0x00007ff6ca900000,0x00007ff6caa00000)]
  0x00007ff7a4004230 JavaThread "Bolt-conn-event-executor-66-thread-1" daemon [_thread_blocked, id=3493975, stack(0x00007ff6ca800000,0x00007ff6ca900000)]
  0x00007ff6ec0031a0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493976, stack(0x00007ff6ca700000,0x00007ff6ca800000)]
  0x00007ff7bc001cd0 JavaThread "RpcTaskScannerThread-70-thread-1" daemon [_thread_blocked, id=3493984, stack(0x00007ff6ca500000,0x00007ff6ca600000)]
  0x00007ff7c0d80530 JavaThread "Bolt-conn-event-executor-69-thread-1" daemon [_thread_blocked, id=3493985, stack(0x00007ff6ca400000,0x00007ff6ca500000)]
  0x00007ff7440054d0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3493986, stack(0x00007ff6ca300000,0x00007ff6ca400000)]
  0x00007ff7d40109e0 JavaThread "RpcTaskScannerThread-73-thread-1" daemon [_thread_blocked, id=3494276, stack(0x00007ff6ca600000,0x00007ff6ca700000)]
  0x00007ff710046660 JavaThread "Bolt-conn-event-executor-72-thread-1" daemon [_thread_blocked, id=3494277, stack(0x00007ff6cecfa000,0x00007ff6cedfa000)]
  0x00007ff728005470 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494282, stack(0x00007ff6d45fe000,0x00007ff6d46fe000)]
  0x00007ff6d8004de0 JavaThread "RpcTaskScannerThread-76-thread-1" daemon [_thread_blocked, id=3494292, stack(0x00007ff6d7e00000,0x00007ff6d7f00000)]
  0x00007ff70001c580 JavaThread "Bolt-conn-event-executor-75-thread-1" daemon [_thread_blocked, id=3494294, stack(0x00007ff7412fe000,0x00007ff7413fe000)]
  0x00007ff77c006900 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494297, stack(0x00007ff6ccffa000,0x00007ff6cd0fa000)]
  0x00007ff6e8007640 JavaThread "RpcTaskScannerThread-79-thread-1" daemon [_thread_blocked, id=3494331, stack(0x00007ff6ce0fa000,0x00007ff6ce1fa000)]
  0x00007ff6e4024be0 JavaThread "Bolt-conn-event-executor-78-thread-1" daemon [_thread_blocked, id=3494333, stack(0x00007ff6cd9fa000,0x00007ff6cdafa000)]
  0x00007ff6e8008660 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494335, stack(0x00007ff6cd6fa000,0x00007ff6cd7fa000)]
  0x00007ff6fc0031b0 JavaThread "RpcTaskScannerThread-82-thread-1" daemon [_thread_blocked, id=3494340, stack(0x00007ff6d01fb000,0x00007ff6d02fb000)]
  0x00007ff70800a8a0 JavaThread "Bolt-conn-event-executor-81-thread-1" daemon [_thread_blocked, id=3494342, stack(0x00007ff6ce4fa000,0x00007ff6ce5fa000)]
  0x00007ff710046ce0 JavaThread "RpcTaskScannerThread-85-thread-1" daemon [_thread_blocked, id=3494346, stack(0x00007ff6d2dfd000,0x00007ff6d2efd000)]
  0x00007ff73c00c700 JavaThread "Bolt-conn-event-executor-84-thread-1" daemon [_thread_blocked, id=3494348, stack(0x00007ff6caa00000,0x00007ff6cab00000)]
  0x00007ff714008ea0 JavaThread "RpcTaskScannerThread-88-thread-1" daemon [_thread_blocked, id=3494358, stack(0x00007ff6cdcfa000,0x00007ff6cddfa000)]
  0x00007ff7900022e0 JavaThread "Bolt-conn-event-executor-87-thread-1" daemon [_thread_blocked, id=3494360, stack(0x00007ff6d60ff000,0x00007ff6d61ff000)]
  0x00007ff728005e20 JavaThread "RpcTaskScannerThread-91-thread-1" daemon [_thread_blocked, id=3494363, stack(0x00007ff6ca200000,0x00007ff6ca300000)]
  0x00007ff7a40051b0 JavaThread "Bolt-conn-event-executor-90-thread-1" daemon [_thread_blocked, id=3494365, stack(0x00007ff6ca100000,0x00007ff6ca200000)]
  0x00007ff72c006400 JavaThread "RpcTaskScannerThread-94-thread-1" daemon [_thread_blocked, id=3494371, stack(0x00007ff6c9f00000,0x00007ff6ca000000)]
  0x00007ff710047dc0 JavaThread "Bolt-conn-event-executor-93-thread-1" daemon [_thread_blocked, id=3494373, stack(0x00007ff6c9e00000,0x00007ff6c9f00000)]
  0x00007ff73c00ec30 JavaThread "RpcTaskScannerThread-97-thread-1" daemon [_thread_blocked, id=3494378, stack(0x00007ff6c9c00000,0x00007ff6c9d00000)]
  0x00007ff70001ce30 JavaThread "Bolt-conn-event-executor-96-thread-1" daemon [_thread_blocked, id=3494380, stack(0x00007ff6c9b00000,0x00007ff6c9c00000)]
  0x00007ff750007080 JavaThread "RpcTaskScannerThread-100-thread-1" daemon [_thread_blocked, id=3494383, stack(0x00007ff6c9900000,0x00007ff6c9a00000)]
  0x00007ff6e400b990 JavaThread "Bolt-conn-event-executor-99-thread-1" daemon [_thread_blocked, id=3494385, stack(0x00007ff6c9800000,0x00007ff6c9900000)]
  0x00007ff744005fa0 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494389, stack(0x00007ff6c9700000,0x00007ff6c9800000)]
  0x00007ff7600062d0 JavaThread "RpcTaskScannerThread-103-thread-1" daemon [_thread_blocked, id=3494391, stack(0x00007ff6c9500000,0x00007ff6c9600000)]
  0x00007ff73c010290 JavaThread "Bolt-conn-event-executor-102-thread-1" daemon [_thread_blocked, id=3494393, stack(0x00007ff6c9400000,0x00007ff6c9500000)]
  0x00007ff778008980 JavaThread "JRaft-RPC-Processor-0" daemon [_thread_blocked, id=3494394, stack(0x00007ff6c9300000,0x00007ff6c9400000)]
...
fengjiachun commented 1 month ago

Did you call RaftGroupService#shutdown when your app stoped?

One more thing to note: if you have enabled the RPC server sharing, you need to close the RPC server separately.

fengjiachun commented 1 month ago

I'm closing it now, if there are any updates, feel free to reopen it.