search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.19k stars 3.58k forks source link

Pulsar EC2 instance out of memory #13858

Open youg9203 opened 2 years ago

youg9203 commented 2 years ago

Describe the bug I have deployed the pulsar on AWS using infrastructure repo, and next, I tried the pulsar command in ec2 instance using ssh and I saw the ec2 instance is out of memory and throws below error:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004c0000000, 12884901888, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12884901888 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid8027.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12884901888 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2795), pid=8027, tid=0x00007fbc9e9e7700
#
# JRE version:  (8.0_312-b07) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.312-b07 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007fbc98055800):  JavaThread "Unknown thread" [_thread_in_vm, id=8041, stack(0x00007fbc9e8e8000,0x00007fbc9e9e8000)]

Stack: [0x00007fbc9e8e8000,0x00007fbc9e9e8000],  sp=0x00007fbc9e9e6420,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xb680cd]  VMError::report_and_die()+0x1cd
V  [libjvm.so+0x508f9a]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0xaa
V  [libjvm.so+0x942b77]  os::pd_commit_memory_or_exit(char*, unsigned long, unsigned long, bool, char const*)+0xe7
V  [libjvm.so+0x93a5bc]  os::commit_memory_or_exit(char*, unsigned long, unsigned long, bool, char const*)+0x2c
V  [libjvm.so+0x5ceac4]  G1PageBasedVirtualSpace::commit_internal(unsigned long, unsigned long)+0x224
V  [libjvm.so+0x5cebc4]  G1PageBasedVirtualSpace::commit(unsigned long, unsigned long)+0xf4
V  [libjvm.so+0x5d13e8]  G1RegionsLargerThanCommitSizeMapper::commit_regions(unsigned int, unsigned long)+0x38
V  [libjvm.so+0x630145]  HeapRegionManager::commit_regions(unsigned int, unsigned long)+0x75
V  [libjvm.so+0x63021d]  HeapRegionManager::make_regions_available(unsigned int, unsigned int)+0x3d
V  [libjvm.so+0x6308d7]  HeapRegionManager::expand_by(unsigned int)+0x67
V  [libjvm.so+0x5a81bd]  G1CollectedHeap::expand(unsigned long)+0x16d
V  [libjvm.so+0x5b2237]  G1CollectedHeap::initialize()+0x7e7
V  [libjvm.so+0xb30b99]  Universe::initialize_heap()+0x159
V  [libjvm.so+0xb30e42]  universe_init()+0x42
V  [libjvm.so+0x645665]  init_globals()+0x65
V  [libjvm.so+0xb12f6f]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x29f
V  [libjvm.so+0x7181fd]  JNI_CreateJavaVM+0x5d
C  [libjli.so+0x3043]  JavaMain+0xa3
C  [libpthread.so.0+0x7ea5]  start_thread+0xc5

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007fbc98055800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=8041, stack(0x00007fbc9e8e8000,0x00007fbc9e9e8000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007fbc98053400] Heap_lock - owner thread: 0x00007fbc98055800

heap address: 0x00000004c0000000, size: 12288 MB, Compressed Oops mode: Non-zero based:0x00000004bffff000, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 0

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events

Dynamic libraries:
558033b01000-558033b02000 r-xp 00000000 103:02 20972394                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/bin/java
558033d01000-558033d02000 r--p 00000000 103:02 20972394                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/bin/java
558033d02000-558033d03000 rw-p 00001000 103:02 20972394                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/bin/java
558034098000-5580340b9000 rw-p 00000000 00:00 0                          [heap]
7fbb5f7f5000-7fbb69af3000 rw-p 00000000 00:00 0
7fbb69af3000-7fbb69af4000 ---p 00000000 00:00 0
7fbb69af4000-7fbb69bf4000 rw-p 00000000 00:00 0
7fbb69bf4000-7fbb69bf5000 ---p 00000000 00:00 0
7fbb69bf5000-7fbb69cf5000 rw-p 00000000 00:00 0
7fbb69cf5000-7fbb69cf6000 ---p 00000000 00:00 0
7fbb69cf6000-7fbb69df6000 rw-p 00000000 00:00 0
7fbb69df6000-7fbb69df7000 ---p 00000000 00:00 0
7fbb69df7000-7fbb69ef7000 rw-p 00000000 00:00 0
7fbb69ef7000-7fbb69ef8000 ---p 00000000 00:00 0
7fbb69ef8000-7fbb69ff8000 rw-p 00000000 00:00 0
7fbb69ff8000-7fbb69ff9000 ---p 00000000 00:00 0
7fbb69ff9000-7fbb6a0f9000 rw-p 00000000 00:00 0
7fbb6a0f9000-7fbb6a0fa000 ---p 00000000 00:00 0
7fbb6a0fa000-7fbb6a1fa000 rw-p 00000000 00:00 0
7fbb6a1fa000-7fbb6a1fb000 ---p 00000000 00:00 0
7fbb6a1fb000-7fbb6a2fb000 rw-p 00000000 00:00 0
7fbb6a2fb000-7fbb6a2fc000 ---p 00000000 00:00 0
7fbb6a2fc000-7fbb6a3fc000 rw-p 00000000 00:00 0
7fbb6a3fc000-7fbb6a3fd000 ---p 00000000 00:00 0
7fbb6a3fd000-7fbb6a4fd000 rw-p 00000000 00:00 0
7fbb6a4fd000-7fbb6a4fe000 ---p 00000000 00:00 0
7fbb6a4fe000-7fbb6a5fe000 rw-p 00000000 00:00 0
7fbb6a5fe000-7fbb6a5ff000 ---p 00000000 00:00 0
7fbb6a5ff000-7fbb6a6ff000 rw-p 00000000 00:00 0
7fbb6a6ff000-7fbb6a700000 ---p 00000000 00:00 0
7fbb6a700000-7fbb6a800000 rw-p 00000000 00:00 0
7fbb6a800000-7fbb84000000 ---p 00000000 00:00 0
7fbb84000000-7fbb84021000 rw-p 00000000 00:00 0
7fbb84021000-7fbb88000000 ---p 00000000 00:00 0
7fbb880f1000-7fbb880f2000 ---p 00000000 00:00 0
7fbb880f2000-7fbb881f2000 rw-p 00000000 00:00 0
7fbb881f2000-7fbb881f3000 ---p 00000000 00:00 0
7fbb881f3000-7fbb882f3000 rw-p 00000000 00:00 0
7fbb882f3000-7fbb882f4000 ---p 00000000 00:00 0
7fbb882f4000-7fbb883f4000 rw-p 00000000 00:00 0
7fbb883f4000-7fbb883f5000 ---p 00000000 00:00 0
7fbb883f5000-7fbb884f5000 rw-p 00000000 00:00 0
7fbb884f5000-7fbb884f6000 ---p 00000000 00:00 0
7fbb884f6000-7fbb885f6000 rw-p 00000000 00:00 0
7fbb885f6000-7fbb885f7000 ---p 00000000 00:00 0
7fbb885f7000-7fbb886f7000 rw-p 00000000 00:00 0
7fbb886f7000-7fbb886f8000 ---p 00000000 00:00 0
7fbb886f8000-7fbb887f8000 rw-p 00000000 00:00 0
7fbb887f8000-7fbb887f9000 ---p 00000000 00:00 0
7fbb887f9000-7fbb888f9000 rw-p 00000000 00:00 0
7fbb888f9000-7fbb888fa000 ---p 00000000 00:00 0
7fbb888fa000-7fbb889fa000 rw-p 00000000 00:00 0
7fbb889fa000-7fbb889fb000 ---p 00000000 00:00 0
7fbb889fb000-7fbb88afb000 rw-p 00000000 00:00 0
7fbb88afb000-7fbb88afc000 ---p 00000000 00:00 0
7fbb88afc000-7fbb88bfc000 rw-p 00000000 00:00 0
7fbb88bfc000-7fbb88bfd000 ---p 00000000 00:00 0
7fbb88bfd000-7fbb88cfd000 rw-p 00000000 00:00 0
7fbb88cfd000-7fbb88cfe000 ---p 00000000 00:00 0
7fbb88cfe000-7fbb88dfe000 rw-p 00000000 00:00 0
7fbb88dfe000-7fbb88dff000 ---p 00000000 00:00 0
7fbb88dff000-7fbb88eff000 rw-p 00000000 00:00 0
7fbb88eff000-7fbb88f00000 ---p 00000000 00:00 0
7fbb88f00000-7fbb89000000 rw-p 00000000 00:00 0
7fbb89000000-7fbb8c000000 ---p 00000000 00:00 0
7fbb8c000000-7fbb8c021000 rw-p 00000000 00:00 0
7fbb8c021000-7fbb90000000 ---p 00000000 00:00 0
7fbb90000000-7fbb90021000 rw-p 00000000 00:00 0
7fbb90021000-7fbb94000000 ---p 00000000 00:00 0
7fbb94000000-7fbb94021000 rw-p 00000000 00:00 0
7fbb94021000-7fbb98000000 ---p 00000000 00:00 0
7fbb98000000-7fbb98021000 rw-p 00000000 00:00 0
7fbb98021000-7fbb9c000000 ---p 00000000 00:00 0
7fbb9c000000-7fbb9c021000 rw-p 00000000 00:00 0
7fbb9c021000-7fbba0000000 ---p 00000000 00:00 0
7fbba0000000-7fbba0021000 rw-p 00000000 00:00 0
7fbba0021000-7fbba4000000 ---p 00000000 00:00 0
7fbba4000000-7fbba4021000 rw-p 00000000 00:00 0
7fbba4021000-7fbba8000000 ---p 00000000 00:00 0
7fbba8000000-7fbba8021000 rw-p 00000000 00:00 0
7fbba8021000-7fbbac000000 ---p 00000000 00:00 0
7fbbac000000-7fbbac021000 rw-p 00000000 00:00 0
7fbbac021000-7fbbb0000000 ---p 00000000 00:00 0
7fbbb0000000-7fbbb0021000 rw-p 00000000 00:00 0
7fbbb0021000-7fbbb4000000 ---p 00000000 00:00 0
7fbbb4000000-7fbbb4021000 rw-p 00000000 00:00 0
7fbbb4021000-7fbbb8000000 ---p 00000000 00:00 0
7fbbb8000000-7fbbb8021000 rw-p 00000000 00:00 0
7fbbb8021000-7fbbbc000000 ---p 00000000 00:00 0
7fbbbc000000-7fbbbc021000 rw-p 00000000 00:00 0
7fbbbc021000-7fbbc0000000 ---p 00000000 00:00 0
7fbbc0000000-7fbbc0021000 rw-p 00000000 00:00 0
7fbbc0021000-7fbbc4000000 ---p 00000000 00:00 0
7fbbc4000000-7fbbc4021000 rw-p 00000000 00:00 0
7fbbc4021000-7fbbc8000000 ---p 00000000 00:00 0
7fbbc8000000-7fbbc8021000 rw-p 00000000 00:00 0
7fbbc8021000-7fbbcc000000 ---p 00000000 00:00 0
7fbbcc000000-7fbbcc021000 rw-p 00000000 00:00 0
7fbbcc021000-7fbbd0000000 ---p 00000000 00:00 0
7fbbd0000000-7fbbd0021000 rw-p 00000000 00:00 0
7fbbd0021000-7fbbd4000000 ---p 00000000 00:00 0
7fbbd4000000-7fbbd4021000 rw-p 00000000 00:00 0
7fbbd4021000-7fbbd8000000 ---p 00000000 00:00 0
7fbbd8000000-7fbbd8021000 rw-p 00000000 00:00 0
7fbbd8021000-7fbbdc000000 ---p 00000000 00:00 0
7fbbdc000000-7fbbdc021000 rw-p 00000000 00:00 0
7fbbdc021000-7fbbe0000000 ---p 00000000 00:00 0
7fbbe0000000-7fbbe0021000 rw-p 00000000 00:00 0
7fbbe0021000-7fbbe4000000 ---p 00000000 00:00 0
7fbbe4000000-7fbbe4021000 rw-p 00000000 00:00 0
7fbbe4021000-7fbbe8000000 ---p 00000000 00:00 0
7fbbe8000000-7fbbe8021000 rw-p 00000000 00:00 0
7fbbe8021000-7fbbec000000 ---p 00000000 00:00 0
7fbbec000000-7fbbec021000 rw-p 00000000 00:00 0
7fbbec021000-7fbbf0000000 ---p 00000000 00:00 0
7fbbf0000000-7fbbf0021000 rw-p 00000000 00:00 0
7fbbf0021000-7fbbf4000000 ---p 00000000 00:00 0
7fbbf4000000-7fbbf4021000 rw-p 00000000 00:00 0
7fbbf4021000-7fbbf8000000 ---p 00000000 00:00 0
7fbbf8000000-7fbbf8021000 rw-p 00000000 00:00 0
7fbbf8021000-7fbbfc000000 ---p 00000000 00:00 0
7fbbfc000000-7fbbfc021000 rw-p 00000000 00:00 0
7fbbfc021000-7fbc00000000 ---p 00000000 00:00 0
7fbc00000000-7fbc00021000 rw-p 00000000 00:00 0
7fbc00021000-7fbc04000000 ---p 00000000 00:00 0
7fbc04000000-7fbc04021000 rw-p 00000000 00:00 0
7fbc04021000-7fbc08000000 ---p 00000000 00:00 0
7fbc08000000-7fbc08021000 rw-p 00000000 00:00 0
7fbc08021000-7fbc0c000000 ---p 00000000 00:00 0
7fbc0c000000-7fbc0c021000 rw-p 00000000 00:00 0
7fbc0c021000-7fbc10000000 ---p 00000000 00:00 0
7fbc10000000-7fbc10021000 rw-p 00000000 00:00 0
7fbc10021000-7fbc14000000 ---p 00000000 00:00 0
7fbc14000000-7fbc14021000 rw-p 00000000 00:00 0
7fbc14021000-7fbc18000000 ---p 00000000 00:00 0
7fbc18000000-7fbc18021000 rw-p 00000000 00:00 0
7fbc18021000-7fbc1c000000 ---p 00000000 00:00 0
7fbc1c000000-7fbc1c021000 rw-p 00000000 00:00 0
7fbc1c021000-7fbc20000000 ---p 00000000 00:00 0
7fbc20000000-7fbc20021000 rw-p 00000000 00:00 0
7fbc20021000-7fbc24000000 ---p 00000000 00:00 0
7fbc24000000-7fbc24021000 rw-p 00000000 00:00 0
7fbc24021000-7fbc28000000 ---p 00000000 00:00 0
7fbc28000000-7fbc28021000 rw-p 00000000 00:00 0
7fbc28021000-7fbc2c000000 ---p 00000000 00:00 0
7fbc2c000000-7fbc2c021000 rw-p 00000000 00:00 0
7fbc2c021000-7fbc30000000 ---p 00000000 00:00 0
7fbc30000000-7fbc30021000 rw-p 00000000 00:00 0
7fbc30021000-7fbc34000000 ---p 00000000 00:00 0
7fbc34000000-7fbc34021000 rw-p 00000000 00:00 0
7fbc34021000-7fbc38000000 ---p 00000000 00:00 0
7fbc38000000-7fbc38021000 rw-p 00000000 00:00 0
7fbc38021000-7fbc3c000000 ---p 00000000 00:00 0
7fbc3c000000-7fbc3c021000 rw-p 00000000 00:00 0
7fbc3c021000-7fbc40000000 ---p 00000000 00:00 0
7fbc40000000-7fbc40021000 rw-p 00000000 00:00 0
7fbc40021000-7fbc44000000 ---p 00000000 00:00 0
7fbc44000000-7fbc44021000 rw-p 00000000 00:00 0
7fbc44021000-7fbc48000000 ---p 00000000 00:00 0
7fbc48000000-7fbc48021000 rw-p 00000000 00:00 0
7fbc48021000-7fbc4c000000 ---p 00000000 00:00 0
7fbc4c000000-7fbc4c021000 rw-p 00000000 00:00 0
7fbc4c021000-7fbc50000000 ---p 00000000 00:00 0
7fbc50000000-7fbc50021000 rw-p 00000000 00:00 0
7fbc50021000-7fbc54000000 ---p 00000000 00:00 0
7fbc54000000-7fbc54021000 rw-p 00000000 00:00 0
7fbc54021000-7fbc58000000 ---p 00000000 00:00 0
7fbc58000000-7fbc58021000 rw-p 00000000 00:00 0
7fbc58021000-7fbc5c000000 ---p 00000000 00:00 0
7fbc5c000000-7fbc5c021000 rw-p 00000000 00:00 0
7fbc5c021000-7fbc60000000 ---p 00000000 00:00 0
7fbc60000000-7fbc60021000 rw-p 00000000 00:00 0
7fbc60021000-7fbc64000000 ---p 00000000 00:00 0
7fbc64000000-7fbc64021000 rw-p 00000000 00:00 0
7fbc64021000-7fbc68000000 ---p 00000000 00:00 0
7fbc68000000-7fbc68021000 rw-p 00000000 00:00 0
7fbc68021000-7fbc6c000000 ---p 00000000 00:00 0
7fbc6c000000-7fbc6c021000 rw-p 00000000 00:00 0
7fbc6c021000-7fbc70000000 ---p 00000000 00:00 0
7fbc700db000-7fbc700dc000 ---p 00000000 00:00 0
7fbc700dc000-7fbc701dc000 rw-p 00000000 00:00 0
7fbc701dc000-7fbc701dd000 ---p 00000000 00:00 0
7fbc701dd000-7fbc702dd000 rw-p 00000000 00:00 0
7fbc702dd000-7fbc702de000 ---p 00000000 00:00 0
7fbc702de000-7fbc703de000 rw-p 00000000 00:00 0
7fbc703de000-7fbc703df000 ---p 00000000 00:00 0
7fbc703df000-7fbc704df000 rw-p 00000000 00:00 0
7fbc704df000-7fbc704e0000 ---p 00000000 00:00 0
7fbc704e0000-7fbc708e1000 rw-p 00000000 00:00 0
7fbc708e1000-7fbc720e1000 ---p 00000000 00:00 0
7fbc720e1000-7fbc720e2000 ---p 00000000 00:00 0
7fbc720e2000-7fbc721e2000 rw-p 00000000 00:00 0
7fbc721e2000-7fbc721e3000 ---p 00000000 00:00 0
7fbc721e3000-7fbc722e3000 rw-p 00000000 00:00 0
7fbc722e3000-7fbc722e4000 ---p 00000000 00:00 0
7fbc722e4000-7fbc723e4000 rw-p 00000000 00:00 0
7fbc723e4000-7fbc723e5000 ---p 00000000 00:00 0
7fbc723e5000-7fbc724e5000 rw-p 00000000 00:00 0
7fbc724e5000-7fbc724e6000 ---p 00000000 00:00 0
7fbc724e6000-7fbc725e6000 rw-p 00000000 00:00 0
7fbc725e6000-7fbc725e7000 ---p 00000000 00:00 0
7fbc725e7000-7fbc726e7000 rw-p 00000000 00:00 0
7fbc726e7000-7fbc726e8000 ---p 00000000 00:00 0
7fbc726e8000-7fbc727e8000 rw-p 00000000 00:00 0
7fbc727e8000-7fbc727e9000 ---p 00000000 00:00 0
7fbc727e9000-7fbc728e9000 rw-p 00000000 00:00 0
7fbc728e9000-7fbc728ea000 ---p 00000000 00:00 0
7fbc728ea000-7fbc729ea000 rw-p 00000000 00:00 0
7fbc729ea000-7fbc729eb000 ---p 00000000 00:00 0
7fbc729eb000-7fbc72aeb000 rw-p 00000000 00:00 0
7fbc72aeb000-7fbc72aec000 ---p 00000000 00:00 0
7fbc72aec000-7fbc72bec000 rw-p 00000000 00:00 0
7fbc72bec000-7fbc72bed000 ---p 00000000 00:00 0
7fbc72bed000-7fbc72ced000 rw-p 00000000 00:00 0
7fbc72ced000-7fbc72cee000 ---p 00000000 00:00 0
7fbc72cee000-7fbc72dee000 rw-p 00000000 00:00 0
7fbc72dee000-7fbc72def000 ---p 00000000 00:00 0
7fbc72def000-7fbc72eef000 rw-p 00000000 00:00 0
7fbc72eef000-7fbc72ef0000 ---p 00000000 00:00 0
7fbc72ef0000-7fbc72ff0000 rw-p 00000000 00:00 0
7fbc72ff0000-7fbc72ff1000 ---p 00000000 00:00 0
7fbc72ff1000-7fbc730f1000 rw-p 00000000 00:00 0
7fbc730f1000-7fbc730f2000 ---p 00000000 00:00 0
7fbc730f2000-7fbc731f2000 rw-p 00000000 00:00 0
7fbc731f2000-7fbc731f3000 ---p 00000000 00:00 0
7fbc731f3000-7fbc732f3000 rw-p 00000000 00:00 0
7fbc732f3000-7fbc732f4000 ---p 00000000 00:00 0
7fbc732f4000-7fbc733f4000 rw-p 00000000 00:00 0
7fbc733f4000-7fbc733f5000 ---p 00000000 00:00 0
7fbc733f5000-7fbc734f5000 rw-p 00000000 00:00 0
7fbc734f5000-7fbc734f6000 ---p 00000000 00:00 0
7fbc734f6000-7fbc735f6000 rw-p 00000000 00:00 0
7fbc735f6000-7fbc735f7000 ---p 00000000 00:00 0
7fbc735f7000-7fbc736f7000 rw-p 00000000 00:00 0
7fbc736f7000-7fbc736f8000 ---p 00000000 00:00 0
7fbc736f8000-7fbc737f8000 rw-p 00000000 00:00 0
7fbc737f8000-7fbc737f9000 ---p 00000000 00:00 0
7fbc737f9000-7fbc738f9000 rw-p 00000000 00:00 0
7fbc738f9000-7fbc738fa000 ---p 00000000 00:00 0
7fbc738fa000-7fbc739fa000 rw-p 00000000 00:00 0
7fbc739fa000-7fbc739fb000 ---p 00000000 00:00 0
7fbc739fb000-7fbc73afb000 rw-p 00000000 00:00 0
7fbc73afb000-7fbc73afc000 ---p 00000000 00:00 0
7fbc73afc000-7fbc73bfc000 rw-p 00000000 00:00 0
7fbc73bfc000-7fbc73bfd000 ---p 00000000 00:00 0
7fbc73bfd000-7fbc73cfd000 rw-p 00000000 00:00 0
7fbc73cfd000-7fbc73cfe000 ---p 00000000 00:00 0
7fbc73cfe000-7fbc73dfe000 rw-p 00000000 00:00 0
7fbc73dfe000-7fbc73dff000 ---p 00000000 00:00 0
7fbc73dff000-7fbc73eff000 rw-p 00000000 00:00 0
7fbc73eff000-7fbc73f00000 ---p 00000000 00:00 0
7fbc73f00000-7fbc74000000 rw-p 00000000 00:00 0
7fbc74000000-7fbc74021000 rw-p 00000000 00:00 0
7fbc74021000-7fbc78000000 ---p 00000000 00:00 0
7fbc780c1000-7fbc780c2000 ---p 00000000 00:00 0
7fbc780c2000-7fbc781c2000 rw-p 00000000 00:00 0
7fbc781c2000-7fbc781c3000 ---p 00000000 00:00 0
7fbc781c3000-7fbc782c3000 rw-p 00000000 00:00 0
7fbc782c3000-7fbc782c4000 ---p 00000000 00:00 0
7fbc782c4000-7fbc7a3e4000 rw-p 00000000 00:00 0
7fbc7a3e4000-7fbc7a3e5000 ---p 00000000 00:00 0
7fbc7a3e5000-7fbc7a4e5000 rw-p 00000000 00:00 0
7fbc7a4e5000-7fbc7a4e6000 ---p 00000000 00:00 0
7fbc7a4e6000-7fbc7a5e6000 rw-p 00000000 00:00 0
7fbc7a5e6000-7fbc7a5e7000 ---p 00000000 00:00 0
7fbc7a5e7000-7fbc7a6e7000 rw-p 00000000 00:00 0
7fbc7a6e7000-7fbc7a6e8000 ---p 00000000 00:00 0
7fbc7a6e8000-7fbc7a7e8000 rw-p 00000000 00:00 0
7fbc7a7e8000-7fbc7a7e9000 ---p 00000000 00:00 0
7fbc7a7e9000-7fbc7a8e9000 rw-p 00000000 00:00 0
7fbc7a8e9000-7fbc7a8ea000 ---p 00000000 00:00 0
7fbc7a8ea000-7fbc7a9ea000 rw-p 00000000 00:00 0
7fbc7a9ea000-7fbc7a9eb000 ---p 00000000 00:00 0
7fbc7a9eb000-7fbc7aaeb000 rw-p 00000000 00:00 0
7fbc7aaeb000-7fbc7aaec000 ---p 00000000 00:00 0
7fbc7aaec000-7fbc7abec000 rw-p 00000000 00:00 0
7fbc7abec000-7fbc7abed000 ---p 00000000 00:00 0
7fbc7abed000-7fbc7aced000 rw-p 00000000 00:00 0
7fbc7aced000-7fbc7acee000 ---p 00000000 00:00 0
7fbc7acee000-7fbc7adee000 rw-p 00000000 00:00 0
7fbc7adee000-7fbc7adef000 ---p 00000000 00:00 0
7fbc7adef000-7fbc7aeef000 rw-p 00000000 00:00 0
7fbc7aeef000-7fbc7aef0000 ---p 00000000 00:00 0
7fbc7aef0000-7fbc7aff0000 rw-p 00000000 00:00 0
7fbc7aff0000-7fbc7aff1000 ---p 00000000 00:00 0
7fbc7aff1000-7fbc7b0f1000 rw-p 00000000 00:00 0
7fbc7b0f1000-7fbc7b0f2000 ---p 00000000 00:00 0
7fbc7b0f2000-7fbc7b1f2000 rw-p 00000000 00:00 0
7fbc7b1f2000-7fbc7b1f3000 ---p 00000000 00:00 0
7fbc7b1f3000-7fbc7b2f3000 rw-p 00000000 00:00 0
7fbc7b2f3000-7fbc7b2f4000 ---p 00000000 00:00 0
7fbc7b2f4000-7fbc7b3f4000 rw-p 00000000 00:00 0
7fbc7b3f4000-7fbc7b3f5000 ---p 00000000 00:00 0
7fbc7b3f5000-7fbc7b4f5000 rw-p 00000000 00:00 0
7fbc7b4f5000-7fbc7b4f6000 ---p 00000000 00:00 0
7fbc7b4f6000-7fbc7b5f6000 rw-p 00000000 00:00 0
7fbc7b5f6000-7fbc7b5f7000 ---p 00000000 00:00 0
7fbc7b5f7000-7fbc7b6f7000 rw-p 00000000 00:00 0
7fbc7b6f7000-7fbc7b6f8000 ---p 00000000 00:00 0
7fbc7b6f8000-7fbc7b7f8000 rw-p 00000000 00:00 0
7fbc7b7f8000-7fbc7b7f9000 ---p 00000000 00:00 0
7fbc7b7f9000-7fbc7b8f9000 rw-p 00000000 00:00 0
7fbc7b8f9000-7fbc7b8fa000 ---p 00000000 00:00 0
7fbc7b8fa000-7fbc7b9fa000 rw-p 00000000 00:00 0
7fbc7b9fa000-7fbc7b9fb000 ---p 00000000 00:00 0
7fbc7b9fb000-7fbc7bafb000 rw-p 00000000 00:00 0
7fbc7bafb000-7fbc7bafc000 ---p 00000000 00:00 0
7fbc7bafc000-7fbc7bbfc000 rw-p 00000000 00:00 0
7fbc7bbfc000-7fbc7bbfd000 ---p 00000000 00:00 0
7fbc7bbfd000-7fbc7bcfd000 rw-p 00000000 00:00 0
7fbc7bcfd000-7fbc7bcfe000 ---p 00000000 00:00 0
7fbc7bcfe000-7fbc7bdfe000 rw-p 00000000 00:00 0
7fbc7bdfe000-7fbc7bdff000 ---p 00000000 00:00 0
7fbc7bdff000-7fbc7beff000 rw-p 00000000 00:00 0
7fbc7beff000-7fbc7bf00000 ---p 00000000 00:00 0
7fbc7bf00000-7fbc7c000000 rw-p 00000000 00:00 0
7fbc7c000000-7fbc7c021000 rw-p 00000000 00:00 0
7fbc7c021000-7fbc80000000 ---p 00000000 00:00 0
7fbc80000000-7fbc80021000 rw-p 00000000 00:00 0
7fbc80021000-7fbc84000000 ---p 00000000 00:00 0
7fbc84000000-7fbc84021000 rw-p 00000000 00:00 0
7fbc84021000-7fbc88000000 ---p 00000000 00:00 0
7fbc880e6000-7fbc880e7000 ---p 00000000 00:00 0
7fbc880e7000-7fbc881e7000 rw-p 00000000 00:00 0
7fbc881e7000-7fbc881e8000 ---p 00000000 00:00 0
7fbc881e8000-7fbc882e8000 rw-p 00000000 00:00 0
7fbc882e8000-7fbc882e9000 ---p 00000000 00:00 0
7fbc882e9000-7fbc883f3000 rw-p 00000000 00:00 0
7fbc883f3000-7fbc887a9000 ---p 00000000 00:00 0
7fbc887a9000-7fbc88a19000 rwxp 00000000 00:00 0
7fbc88a19000-7fbc977a9000 ---p 00000000 00:00 0
7fbc977a9000-7fbc977b1000 r-xp 00000000 103:02 5605                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libzip.so
7fbc977b1000-7fbc979b0000 ---p 00008000 103:02 5605                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libzip.so
7fbc979b0000-7fbc979b1000 r--p 00007000 103:02 5605                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libzip.so
7fbc979b1000-7fbc979b2000 rw-p 00008000 103:02 5605                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libzip.so
7fbc979b2000-7fbc979be000 r-xp 00000000 103:02 88393                     /usr/lib64/libnss_files-2.17.so
7fbc979be000-7fbc97bbd000 ---p 0000c000 103:02 88393                     /usr/lib64/libnss_files-2.17.so
7fbc97bbd000-7fbc97bbe000 r--p 0000b000 103:02 88393                     /usr/lib64/libnss_files-2.17.so
7fbc97bbe000-7fbc97bbf000 rw-p 0000c000 103:02 88393                     /usr/lib64/libnss_files-2.17.so
7fbc97bbf000-7fbc97bc5000 rw-p 00000000 00:00 0
7fbc97bc5000-7fbc97bef000 r-xp 00000000 103:02 5587                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libjava.so
7fbc97bef000-7fbc97dee000 ---p 0002a000 103:02 5587                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libjava.so
7fbc97dee000-7fbc97def000 r--p 00029000 103:02 5587                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libjava.so
7fbc97def000-7fbc97df1000 rw-p 0002a000 103:02 5587                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libjava.so
7fbc97df1000-7fbc97dfe000 r-xp 00000000 103:02 5604                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libverify.so
7fbc97dfe000-7fbc97ffd000 ---p 0000d000 103:02 5604                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libverify.so
7fbc97ffd000-7fbc97fff000 r--p 0000c000 103:02 5604                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libverify.so
7fbc97fff000-7fbc98000000 rw-p 0000e000 103:02 5604                      /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/libverify.so
7fbc98000000-7fbc98210000 rw-p 00000000 00:00 0
7fbc98210000-7fbc9c000000 ---p 00000000 00:00 0
7fbc9c074000-7fbc9c0ed000 rw-p 00000000 00:00 0
7fbc9c0ed000-7fbc9c0f4000 r-xp 00000000 103:02 88405                     /usr/lib64/librt-2.17.so
7fbc9c0f4000-7fbc9c2f3000 ---p 00007000 103:02 88405                     /usr/lib64/librt-2.17.so
7fbc9c2f3000-7fbc9c2f4000 r--p 00006000 103:02 88405                     /usr/lib64/librt-2.17.so
7fbc9c2f4000-7fbc9c2f5000 rw-p 00007000 103:02 88405                     /usr/lib64/librt-2.17.so
7fbc9c2f5000-7fbc9c30a000 r-xp 00000000 103:02 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fbc9c30a000-7fbc9c509000 ---p 00015000 103:02 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fbc9c509000-7fbc9c50a000 r--p 00014000 103:02 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fbc9c50a000-7fbc9c50b000 rw-p 00015000 103:02 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fbc9c50b000-7fbc9c60c000 r-xp 00000000 103:02 88383                     /usr/lib64/libm-2.17.so
7fbc9c60c000-7fbc9c80b000 ---p 00101000 103:02 88383                     /usr/lib64/libm-2.17.so
7fbc9c80b000-7fbc9c80c000 r--p 00100000 103:02 88383                     /usr/lib64/libm-2.17.so
7fbc9c80c000-7fbc9c80d000 rw-p 00101000 103:02 88383                     /usr/lib64/libm-2.17.so
7fbc9c80d000-7fbc9c8f6000 r-xp 00000000 103:02 88438                     /usr/lib64/libstdc++.so.6.0.19
7fbc9c8f6000-7fbc9caf5000 ---p 000e9000 103:02 88438                     /usr/lib64/libstdc++.so.6.0.19
7fbc9caf5000-7fbc9cafd000 r--p 000e8000 103:02 88438                     /usr/lib64/libstdc++.so.6.0.19
7fbc9cafd000-7fbc9caff000 rw-p 000f0000 103:02 88438                     /usr/lib64/libstdc++.so.6.0.19
7fbc9caff000-7fbc9cb14000 rw-p 00000000 00:00 0
7fbc9cb14000-7fbc9d8b3000 r-xp 00000000 103:02 12584398                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so
7fbc9d8b3000-7fbc9dab2000 ---p 00d9f000 103:02 12584398                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so
7fbc9dab2000-7fbc9db60000 r--p 00d9e000 103:02 12584398                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so
7fbc9db60000-7fbc9db8b000 rw-p 00e4c000 103:02 12584398                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so
7fbc9db8b000-7fbc9dbc1000 rw-p 00000000 00:00 0
7fbc9dbc1000-7fbc9dd84000 r-xp 00000000 103:02 88375                     /usr/lib64/libc-2.17.so
7fbc9dd84000-7fbc9df84000 ---p 001c3000 103:02 88375                     /usr/lib64/libc-2.17.so
7fbc9df84000-7fbc9df88000 r--p 001c3000 103:02 88375                     /usr/lib64/libc-2.17.so
7fbc9df88000-7fbc9df8a000 rw-p 001c7000 103:02 88375                     /usr/lib64/libc-2.17.so
7fbc9df8a000-7fbc9df8f000 rw-p 00000000 00:00 0
7fbc9df8f000-7fbc9df91000 r-xp 00000000 103:02 88381                     /usr/lib64/libdl-2.17.so
7fbc9df91000-7fbc9e191000 ---p 00002000 103:02 88381                     /usr/lib64/libdl-2.17.so
7fbc9e191000-7fbc9e192000 r--p 00002000 103:02 88381                     /usr/lib64/libdl-2.17.so
7fbc9e192000-7fbc9e193000 rw-p 00003000 103:02 88381                     /usr/lib64/libdl-2.17.so
7fbc9e193000-7fbc9e1a1000 r-xp 00000000 103:02 8410127                   /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/jli/libjli.so
7fbc9e1a1000-7fbc9e3a0000 ---p 0000e000 103:02 8410127                   /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/jli/libjli.so
7fbc9e3a0000-7fbc9e3a1000 r--p 0000d000 103:02 8410127                   /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/jli/libjli.so
7fbc9e3a1000-7fbc9e3a2000 rw-p 0000e000 103:02 8410127                   /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/jli/libjli.so
7fbc9e3a2000-7fbc9e3b7000 r-xp 00000000 103:02 124995                    /usr/lib64/libz.so.1.2.7
7fbc9e3b7000-7fbc9e5b6000 ---p 00015000 103:02 124995                    /usr/lib64/libz.so.1.2.7
7fbc9e5b6000-7fbc9e5b7000 r--p 00014000 103:02 124995                    /usr/lib64/libz.so.1.2.7
7fbc9e5b7000-7fbc9e5b8000 rw-p 00015000 103:02 124995                    /usr/lib64/libz.so.1.2.7
7fbc9e5b8000-7fbc9e5cf000 r-xp 00000000 103:02 88401                     /usr/lib64/libpthread-2.17.so
7fbc9e5cf000-7fbc9e7ce000 ---p 00017000 103:02 88401                     /usr/lib64/libpthread-2.17.so
7fbc9e7ce000-7fbc9e7cf000 r--p 00016000 103:02 88401                     /usr/lib64/libpthread-2.17.so
7fbc9e7cf000-7fbc9e7d0000 rw-p 00017000 103:02 88401                     /usr/lib64/libpthread-2.17.so
7fbc9e7d0000-7fbc9e7d4000 rw-p 00000000 00:00 0
7fbc9e7d4000-7fbc9e7f6000 r-xp 00000000 103:02 88368                     /usr/lib64/ld-2.17.so
7fbc9e83d000-7fbc9e8df000 rw-p 00000000 00:00 0
7fbc9e8df000-7fbc9e8e7000 rw-s 00000000 103:02 8657576                   /tmp/hsperfdata_ssm-user/8027
7fbc9e8e7000-7fbc9e8eb000 ---p 00000000 00:00 0
7fbc9e8eb000-7fbc9e9ed000 rw-p 00000000 00:00 0
7fbc9e9ef000-7fbc9e9f3000 rw-p 00000000 00:00 0
7fbc9e9f3000-7fbc9e9f4000 r--p 00000000 00:00 0
7fbc9e9f4000-7fbc9e9f5000 rw-p 00000000 00:00 0
7fbc9e9f5000-7fbc9e9f6000 r--p 00021000 103:02 88368                     /usr/lib64/ld-2.17.so
7fbc9e9f6000-7fbc9e9f7000 rw-p 00022000 103:02 88368                     /usr/lib64/ld-2.17.so
7fbc9e9f7000-7fbc9e9f8000 rw-p 00000000 00:00 0
7ffef2733000-7ffef2754000 rw-p 00000000 00:00 0                          [stack]
7ffef2787000-7ffef2789000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Dio.netty.tryReflectionSetAccessible=true -Xms12g -Xmx12g -XX:MaxDirectMemorySize=12g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024 -Xms12g-Xmx12g -XX:MaxDirectMemorySize=12g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -Dpulsar.log.appender=RoutingAppender -Dpulsar.log.dir=/opt/pulsar/logs -Dpulsar.log.level=info -Dpulsar.log.root.level=info -Dpulsar.routing.appender.default=Console -Dlog4j2.is.webapp=false -Dpulsar.functions.process.container.log.dir=/opt/pulsar/logs -Dpulsar.functions.java.instance.jar=/opt/pulsar/instances/java-instance.jar -Dpulsar.functions.python.instance.file=/opt/pulsar/instances/python-instance/python_instance_main.py -Dpulsar.functions.extra.dependencies.dir=/opt/pulsar/instances/deps -Dpulsar.functions.instance.classpath=/opt/pulsar/conf:::/opt/pulsar/lib/*: -Dpulsar.functions.log.conf=/opt/pulsar/conf/functions_log4j2.xml
java_command: org.apache.pulsar.utils.auth.tokens.TokensCliUtils create --secret-key file:///opt/pulsar/pulsar-dev-secret.key --subject test-user
java_class_path (initial): /opt/pulsar/conf:::/opt/pulsar/lib/org.apache.pulsar-pulsar-broker-2.9.1.jar:/opt/pulsar/lib/commons-codec-commons-codec-1.15.jar:/opt/pulsar/lib/commons-collections-commons-collections-3.2.2.jar:/opt/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/opt/pulsar/lib/org.slf4j-slf4j-api-1.7.25.jar:/opt/pulsar/lib/io.netty-netty-transport-4.1.72.Final.jar:/opt/pulsar/lib/com.google.protobuf-protobuf-java-3.11.4.jar:/opt/pulsar/lib/org.apache.pulsar-pulsar-client-original-2.9.1.jar:/opt/pulsar/lib/org.apache.pulsar-pulsar-client-api-2.9.1.jar:/opt/pulsar/lib/io.netty-netty-codec-socks-4.1.72.Final.jar:/opt/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/opt/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/opt/pulsar/lib/org.apache.pulsar-pulsar-websocket-2.9.1.jar:/opt/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.43.v20210629.jar:/opt/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.43.v20210629.jar:/opt/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.43.v20210629.jar:/opt/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.43.v20210629.jar:/opt/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.43.v20210629.jar:/opt/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.43.v20210629.jar:/opt/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/opt/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-2.9.1.jar:/opt/pulsar/lib/org.apache.pulsar-managed-ledger-2.9.1.jar:/opt/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.14.2.jar:/opt/pulsar/lib/org.apache.pulsar-pulsar-metadata-2.9.1.jar:/opt/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.14.2.jar:/opt/pulsar/lib/org.apache.pulsar-pulsar-broker-common-2.9.1.jar:/opt/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/opt/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/opt/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/opt/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

Signal Handlers:
SIGSEGV: [libjvm.so+0xb68b30], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xb68b30], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x93dbe0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x93dbe0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x93dbe0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x93dbe0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x93da90], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none

---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux Server release 7.7 (Maipo)

uname:Linux 3.10.0-1062.4.3.el7.x86_64 #1 SMP Tue Nov 12 10:42:40 EST 2019 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 61498, NOFILE 4096, AS infinity
load average:0.00 0.01 0.05

/proc/meminfo:
MemTotal:       15790548 kB
MemFree:         7797056 kB
MemAvailable:    9271176 kB
Buffers:            2600 kB
Cached:          1701804 kB
SwapCached:            0 kB
Active:          6989852 kB
Inactive:         672160 kB
Active(anon):    5959148 kB
Inactive(anon):    23420 kB
Active(file):    1030704 kB
Inactive(file):   648740 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:       5955904 kB
Mapped:            89160 kB
Shmem:             24952 kB
Slab:             143732 kB
SReclaimable:      93216 kB
SUnreclaim:        50516 kB
KernelStack:        7856 kB
PageTables:        16532 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     7895272 kB
Committed_AS:   15116324 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       36136 kB
VmallocChunk:   34359696292 kB
HardwareCorrupted:     0 kB
AnonHugePages:   5703680 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      116644 kB
DirectMap2M:     3596288 kB
DirectMap1G:    12582912 kB

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-7
cpu_memory_nodes: 0
active_processor_count: 8
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -1
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 7846436864
memory_max_usage_in_bytes: 0

CPU:total 8 (initial active 8) (4 cores per cpu, 2 threads per core) family 6 model 85 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx

/proc/cpuinfo:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 1
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 2
cpu cores       : 4
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 1
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 2
cpu cores       : 4
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
stepping        : 7
microcode       : 0x5003103
cpu MHz         : 2999.998
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke
bogomips        : 5999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Memory: 4k page, physical 15790548k(7797056k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (25.312-b07) for linux-amd64 JRE (1.8.0_312-b07), built on Oct 15 2021 04:33:40 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-44)

time: Thu Jan 20 15:47:15 2022
timezone: UTC
elapsed time: 0.073127 seconds (0d 0h 0m 0s)

I see the issue is swap memory but after restarting the instance swap is still 0k, and the issue is gone. This issue I have seen on the proxy (c5.2xlarge) and broker (c5.2xlarge) instances.

Expected behavior Should not go out of memory.

Desktop (please complete the following information):

Additional context I know add the swapfile solve the problem, but its not getting clear and that I believe not the right solution.

dave2wave commented 2 years ago

Did you use the terraform-aws setup at https://github.com/apache/pulsar/tree/master/deployment/terraform-ansible ?

If so that has been updated in the last few days. You can try to update.

Also you can control memory in:

./deploy-pulsar.yaml
50:        max_heap_memory: "12g"
51:        max_direct_memory: "12g"
65:        max_heap_memory: "512m"
66:        max_direct_memory: "1g"
youg9203 commented 2 years ago

Yes, I have used terraform-aws, and these configurations are already in my deploy-pulsar.xml file.

github-actions[bot] commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.

github-actions[bot] commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.