search

adoptium / adoptium-support

For end-user problems reported with our binary distributions
Apache License 2.0
46 stars 15 forks source link

JVM Crash on Amazon Linux 6.1 #1022

Closed aalwadi-wcq closed 5 months ago

aalwadi-wcq commented 8 months ago

Please provide a brief summary of the bug

Hey Team, We have been having an issue with JVM startup on our Control-M system which uses OpenJDK 11.0.21+9. JVM crashes immediately after startup. The issue happens in the native code, particularly on libstdc++.so.6+0xac96c (which points to /usr/lib64/libstdc++.so.6.0.29). The JVM system is Control-M system running on version 9.0.21.200. Please see below the hs_error file `

A fatal error has been detected by the Java Runtime Environment:

#

SIGSEGV (0xb) at pc=0x00007f78678ac96c, pid=125270, tid=125270

#

JRE version: OpenJDK Runtime Environment Temurin-11.0.21+9 (11.0.21+9) (build 11.0.21+9)

Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.21+9 (11.0.21+9, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)

Problematic frame:

C [libstdc++.so.6+0xac96c] __gxx_personality_v0+0x22c

#

Core dump will be written. Default location: Core dumps may be processed with "/bin/false" (or dumping to /control/ctmem/ctm_em/log/bimsrv/core.125270)

#

If you would like to submit a bug report, please visit:

https://github.com/adoptium/adoptium-support/issues

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

#

--------------- S U M M A R Y ------------

Command Line: -Xmx600m -XX:+CrashOnOutOfMemoryError abort exit vfprintf -XX:OnOutOfMemoryError=/control/ctmem/ctm_em/bin/heapMemErr BIM 600 0 5 300 125270 -Djava.security.egd=file:/dev/./urandom

Host: AMD EPYC 7R13 Processor, 4 cores, 30G, Amazon Linux release 2023 (Amazon Linux) Time: Mon Feb 12 00:24:19 2024 UTC elapsed time: 0.487046 seconds (0d 0h 0m 0s)

--------------- T H R E A D ---------------

Current thread (0x00000000025ec000): JavaThread "main" [_thread_in_native, id=125270, stack(0x00007ffef7b20000,0x00007ffef7c20000)]

Stack: [0x00007ffef7b20000,0x00007ffef7c20000], sp=0x00007ffef7c1dbd0, free space=1014k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libstdc++.so.6+0xac96c] __gxx_personality_v0+0x22c C 0x00007ffef7c1e020

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

Register to memory mapping:

RAX=0x0000000000000006 is an unknown value RBX=0x0000000001ec0868 points into unknown readable memory: 0x474e5543432b2b00 | 00 2b 2b 43 43 55 4e 47 RCX=0x0000000000000001 is an unknown value RDX=0x0000000001ec0888 points into unknown readable memory: 0x00007f78768a4350 | 50 43 8a 76 78 7f 00 00 RSP=0x00007ffef7c1dbd0 is pointing into the stack for thread: 0x00000000025ec000 RBP=0x00007ffef7c1dda0 is pointing into the stack for thread: 0x00000000025ec000 RSI=0x00007f787668a6a3: _ZTS8CTDError+0x0000000000000000 in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 RDI=0x00007f787668a6a3: _ZTS8CTDError+0x0000000000000000 in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 R8 =0x00007f78766a31f3: <offset 0x00000000000a31f3> in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 R9 =0x00007f78766a31f8: <offset 0x00000000000a31f8> in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 R10=0x0000000000001c00 is an unknown value R11=0x0000000000000038 is an unknown value R12=0x00007f78768a4370: _ZTI8CTDError+0x0000000000000000 in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 R13=0x0000000000000002 is an unknown value R14=0xfffffffffffffffd is an unknown value R15=0x00007f7876685061: _ZN24Osl_UnicodeConfiguration13ReadLocaleIniEv+0x0000000000001c4b in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000

Registers: RAX=0x0000000000000006, RBX=0x0000000001ec0868, RCX=0x0000000000000001, RDX=0x0000000001ec0888 RSP=0x00007ffef7c1dbd0, RBP=0x00007ffef7c1dda0, RSI=0x00007f787668a6a3, RDI=0x00007f787668a6a3 R8 =0x00007f78766a31f3, R9 =0x00007f78766a31f8, R10=0x0000000000001c00, R11=0x0000000000000038 R12=0x00007f78768a4370, R13=0x0000000000000002, R14=0xfffffffffffffffd, R15=0x00007f7876685061 RIP=0x00007f78678ac96c, EFLAGS=0x0000000000010297, CSGSFS=0x002b000000000033, ERR=0x0000000000000000 TRAPNO=0x000000000000000d

Top of Stack: (sp=0x00007ffef7c1dbd0) 0x00007ffef7c1dbd0: 0000000001ec0868 0000000101683738 0x00007ffef7c1dbe0: 00007f78766a2dd4 0000000000000000 0x00007ffef7c1dbf0: 00007f7876685061 00007ffef7c1dc40 0x00007ffef7c1dc00: 00007ffe00000000 00007ffef7c1dda0

Instructions: (pc=0x00007f78678ac96c) 0x00007f78678ac86c: 4c 8d 74 24 60 4c 8d 6c 24 68 66 2e 0f 1f 84 00 0x00007f78678ac87c: 00 00 00 00 0f b6 ac 24 99 00 00 00 31 f6 89 ef 0x00007f78678ac88c: e8 5f fd ff ff 4c 89 e2 4c 89 f9 89 ef 48 89 c6 0x00007f78678ac89c: e8 cf fa ff ff 0f b6 ac 24 99 00 00 00 31 f6 49 0x00007f78678ac8ac: 89 c4 89 ef e8 3b fd ff ff 4c 89 e2 4c 89 f1 89 0x00007f78678ac8bc: ef 48 89 c6 e8 ab fa ff ff 0f b6 ac 24 99 00 00 0x00007f78678ac8cc: 00 31 f6 49 89 c4 89 ef e8 17 fd ff ff 4c 89 e9 0x00007f78678ac8dc: 4c 89 e2 89 ef 48 89 c6 e8 87 fa ff ff 31 ff 31 0x00007f78678ac8ec: c9 49 89 c4 41 0f b6 14 24 49 83 c4 01 48 89 d0 0x00007f78678ac8fc: 83 e0 7f 48 d3 e0 83 c1 07 48 09 c7 84 d2 78 e4 0x00007f78678ac90c: 48 8b 44 24 58 48 03 44 24 70 48 39 d8 0f 86 b1 0x00007f78678ac91c: 00 00 00 f6 44 24 0c 01 0f 84 c6 00 00 00 45 31 0x00007f78678ac92c: ff 45 31 c0 45 31 ed 48 83 7c 24 18 01 66 49 0f 0x00007f78678ac93c: 6e c0 b8 06 00 00 00 66 48 0f 3a 22 44 24 10 01 0x00007f78678ac94c: 0f 87 83 fe ff ff 48 8b 1c 24 48 8b 44 24 50 44 0x00007f78678ac95c: 89 6b dc 48 89 43 f8 b8 06 00 00 00 4c 89 7b f0 0x00007f78678ac96c: 0f 29 43 e0 e9 60 fe ff ff 45 85 ed 0f 88 a1 00 0x00007f78678ac97c: 00 00 66 90 48 8b 14 24 31 f6 48 89 ef e8 22 0d 0x00007f78678ac98c: ff ff 49 63 d5 be 01 00 00 00 48 89 ef e8 12 0d 0x00007f78678ac99c: ff ff 4c 89 fe 48 89 ef e8 87 46 ff ff b8 07 00 0x00007f78678ac9ac: 00 00 e9 22 fe ff ff 4c 8b 7c 24 68 48 8b 6c 24 0x00007f78678ac9bc: 30 4d 85 ff 75 3e b8 08 00 00 00 e9 09 fe ff ff 0x00007f78678ac9cc: 0f 1f 40 00 48 03 44 24 60 48 39 d8 77 d9 4c 39 0x00007f78678ac9dc: a4 24 90 00 00 00 0f 87 98 fe ff ff e9 32 ff ff 0x00007f78678ac9ec: ff 0f 1f 00 f6 44 24 0c 08 74 6f e8 a4 46 ff ff 0x00007f78678ac9fc: 0f 1f 40 00 4c 03 7c 24 78 48 85 ff 75 6e 4d 85 0x00007f78678aca0c: ff 74 b3 44 8b 6c 24 0c 41 83 e5 01 75 a8 e9 61 0x00007f78678aca1c: ff ff ff 48 8b 74 24 10 48 8d 54 24 70 48 89 ef 0x00007f78678aca2c: e8 2f fc ff ff 0f b6 bc 24 98 00 00 00 48 89 ee 0x00007f78678aca3c: e8 af fb ff ff 0f b6 bc 24 98 00 00 00 48 89 ee 0x00007f78678aca4c: 48 89 84 24 80 00 00 00 e8 97 fb ff ff 48 8b 1c 0x00007f78678aca5c: 24 48 89 43 f0 e9 1a ff ff ff 48 83 7c 24 18 01

Stack slot to memory mapping: stack at sp + 0 slots: 0x0000000001ec0868 points into unknown readable memory: 0x474e5543432b2b00 | 00 2b 2b 43 43 55 4e 47 stack at sp + 1 slots: 0x0000000101683738 is an unknown value stack at sp + 2 slots: 0x00007f78766a2dd4: <offset 0x00000000000a2dd4> in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 stack at sp + 3 slots: 0x0 is NULL stack at sp + 4 slots: 0x00007f7876685061: _ZN24Osl_UnicodeConfiguration13ReadLocaleIniEv+0x0000000000001c4b in /control/ctmem/ctm_em/lib/libbmc-corekit.so.4.0 at 0x00007f7876600000 stack at sp + 5 slots: 0x00007ffef7c1dc40 is pointing into the stack for thread: 0x00000000025ec000 stack at sp + 6 slots: 0x00007ffe00000000 is an unknown value stack at sp + 7 slots: 0x00007ffef7c1dda0 is pointing into the stack for thread: 0x00000000025ec000

--------------- P R O C E S S ---------------

Threads class SMR info: _java_thread_list=0x0000000003e2db58, length=11, elements={ 0x00000000025ec000, 0x0000000003332800, 0x000000000333c000, 0x00000000033b1000, 0x00000000032b5800, 0x00000000032b4000, 0x00000000032b2000, 0x00000000032c0800, 0x0000000003473800, 0x00000000032f1800, 0x0000000003f01800 }

Java Threads: ( => current thread ) =>0x00000000025ec000 JavaThread "main" [_thread_in_native, id=125270, stack(0x00007ffef7b20000,0x00007ffef7c20000)] 0x0000000003332800 JavaThread "Reference Handler" daemon [_thread_blocked, id=125277, stack(0x00007f7873700000,0x00007f7873800000)] 0x000000000333c000 JavaThread "Finalizer" daemon [_thread_blocked, id=125278, stack(0x00007f7873300000,0x00007f7873400000)] 0x00000000033b1000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=125279, stack(0x00007f7872f00000,0x00007f7873000000)] 0x00000000032b5800 JavaThread "Service Thread" daemon [_thread_blocked, id=125280, stack(0x00007f7872b00000,0x00007f7872c00000)] 0x00000000032b4000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=125281, stack(0x00007f7872700000,0x00007f7872800000)] 0x00000000032b2000 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=125282, stack(0x00007f7872300000,0x00007f7872400000)] 0x00000000032c0800 JavaThread "Sweeper thread" daemon [_thread_blocked, id=125283, stack(0x00007f7871f00000,0x00007f7872000000)] 0x0000000003473800 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=125285, stack(0x00007f7870700000,0x00007f7870800000)] 0x00000000032f1800 JavaThread "Tomcat JDBC Pool Cleaner[258754732:1707697459553]" daemon [_thread_blocked, id=125286, stack(0x00007f7870300000,0x00007f7870400000)] 0x0000000003f01800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=125287, stack(0x00007f786fd00000,0x00007f786fe00000)]

Other Threads: 0x00000000032f7800 VMThread "VM Thread" [stack: 0x00007f7874100000,0x00007f7874200000] [id=125276] 0x0000000003455000 WatcherThread [stack: 0x00007f7871500000,0x00007f7871600000] [id=125284] 0x0000000002619800 GCTaskThread "GC Thread#0" [stack: 0x00007f7878500000,0x00007f7878600000] [id=125271] 0x0000000002a96800 ConcurrentGCThread "G1 Main Marker" [stack: 0x00007f7877500000,0x00007f7877600000] [id=125272] 0x0000000002a95000 ConcurrentGCThread "G1 Conc#0" [stack: 0x00007f7877100000,0x00007f7877200000] [id=125273] 0x00000000030be800 ConcurrentGCThread "G1 Refine#0" [stack: 0x00007f7876d00000,0x00007f7876e00000] [id=125274] 0x00000000030bd000 ConcurrentGCThread "G1 Young RemSet Sampling" [stack: 0x00007f7876100000,0x00007f7876200000] [id=125275]

Threads with active compile tasks:

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

--------------- S Y S T E M ---------------

OS:Amazon Linux release 2023 (Amazon Linux) uname:Linux 6.1.61-85.141.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8 00:39:18 UTC 2023 x86_64 OS uptime: 0 days 6:01 hours libc:glibc 2.34 NPTL 2.34 rlimit (soft/hard): STACK 8192k/10240k , CORE 0k/0k , NPROC infinity/infinity , NOFILE 65535/65535 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK infinity/infinity load average:0.39 0.25 0.26

/proc/meminfo: MemTotal: 32132332 kB MemFree: 18735904 kB MemAvailable: 24301312 kB Buffers: 2168 kB Cached: 5737588 kB SwapCached: 0 kB Active: 4619756 kB Inactive: 8198368 kB Active(anon): 1028 kB Inactive(anon): 7086680 kB Active(file): 4618728 kB Inactive(file): 1111688 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Zswap: 0 kB Zswapped: 0 kB Dirty: 528 kB Writeback: 0 kB AnonPages: 7055540 kB Mapped: 496212 kB Shmem: 9340 kB KReclaimable: 294924 kB Slab: 373368 kB SReclaimable: 294924 kB SUnreclaim: 78444 kB KernelStack: 18472 kB PageTables: 29244 kB SecPageTables: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 16066164 kB Committed_AS: 11380812 kB VmallocTotal: 34359738367 kB VmallocUsed: 26780 kB VmallocChunk: 0 kB Percpu: 2048 kB HardwareCorrupted: 0 kB AnonHugePages: 8192 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 98736 kB DirectMap2M: 5455872 kB DirectMap1G: 28311552 kB

/sys/kernel/mm/transparent_hugepage/enabled: always [madvise] never /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter): always defer defer+madvise [madvise] never

Process Memory: Virtual Size: 2389044K (peak: 2389048K) Resident Set Size: 131140K (peak: 131140K) (anon: 60840K, file: 70268K, shmem: 32K) Swapped out: 0K C-Heap outstanding allocations: 0K, retained: 0K glibc malloc tunables: (default)

/proc/sys/kernel/threads-max (system-wide limit on the number of threads): 250870 /proc/sys/vm/max_map_count (maximum number of memory map areas a process may have): 65530 /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers): 4194304

container (cgroup) information: container_type: cgroupv2 cpu_cpuset_cpus: not supported cpu_memory_nodes: not supported active_processor_count: 4 cpu_quota: no quota cpu_period: 100000 cpu_shares: no shares memory_limit_in_bytes: unlimited memory_and_swap_limit_in_bytes: unlimited memory_soft_limit_in_bytes: unlimited memory_usage_in_bytes: 11921336 k memory_max_usage_in_bytes: not supported memory_swap_current_in_bytes: unlimited memory_swap_max_limit_in_bytes: unlimited maximum number of tasks: unlimited current number of tasks: 947

KVM virtualization detected Steal ticks since vm start: 0 Steal ticks percentage since vm start: 0.000

CPU:total 4 (initial active 4) (4 cores per cpu, 2 threads per core) family 25 model 1 stepping 1 microcode 0xa0011d1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, ht, tsc, tscinvbit, tscinv, bmi1, bmi2, adx, sha, fma CPU Model and flags from /proc/cpuinfo: model name : AMD EPYC 7R13 Processor flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single pti ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid Online cpus: 0-3 Offline cpus:

BIOS frequency limitation:

Frequency switch latency (ns): Available cpu frequencies: container (cgroup) information: container_type: cgroupv2 cpu_cpuset_cpus: not supported cpu_memory_nodes: not supported active_processor_count: 4 cpu_quota: no quota cpu_period: 100000 cpu_shares: no shares memory_limit_in_bytes: unlimited memory_and_swap_limit_in_bytes: unlimited memory_soft_limit_in_bytes: unlimited memory_usage_in_bytes: 11921336 k memory_max_usage_in_bytes: not supported memory_swap_current_in_bytes: unlimited memory_swap_max_limit_in_bytes: unlimited maximum number of tasks: unlimited current number of tasks: 947 KVM virtualization detected Steal ticks since vm start: 0 Steal ticks percentage since vm start: 0.000 CPU:total 4 (initial active 4) (4 cores per cpu, 2 threads per core) family 25 model 1 stepping 1 microcode 0xa0011d1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, ht, tsc, tscinvbit, tscinv, bmi1, bmi2, adx, sha, fma CPU Model and flags from /proc/cpuinfo: model name : AMD EPYC 7R13 Processor flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single pti ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid Online cpus: 0-3 Offline cpus: BIOS frequency limitation: Frequency switch latency (ns): Available cpu frequencies: Current governor: Core performance/turbo boost: Memory: 4k page, physical 32132332k(18735904k free), swap 0k(0k free) vm_info: OpenJDK 64-Bit Server VM (11.0.21+9) for linux-amd64 JRE (11.0.21+9), built on Oct 17 2023 21:44:42 by "" with gcc 7.5.0 END. ` OS Version: Linux aws-ctmem-01 6.1.61-85.141.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8 00:39:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ### Did you test with the latest update version? ```Shell Yes. This was tested on multiple JDK versions, include: 11.0.16, 11.0.21+9, and 11.0.21+13 ``` ### Please provide steps to reproduce where possible _No response_ ### Expected Results JVM startup completes and JVM is up and running ### Actual Results JVM startup failure ### What Java Version are you using? openjdk 11.0.21 2023-10-17 OpenJDK Runtime Environment Temurin-11.0.21+9 (build 11.0.21+9) OpenJDK 64-Bit Server VM Temurin-11.0.21+9 (build 11.0.21+9, mixed mode) ### What is your operating system and platform? Amazon Linux : Linux aws-ctmem-01 6.1.61-85.141.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8 00:39:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ### How did you install Java? Downloading the JDK/JRE from https://adoptium.net/en-GB/download/ . untar the file and copy to the required path. Finally, executing update-alternatives to update the system alternatives. ### Did it work before? ```Shell NA. This has never been tested before on the same OS using the same JDK version ``` ### Did you test with other Java versions? _No response_ ### Relevant log output _No response_
karianna commented 8 months ago

@aalwadi-wcq This is a crash in native code that happens to propagate back through the JVM. Are you able to post this bug to the author of the CTM software?

aalwadi-wcq commented 8 months ago

Hi Karianna,

Thank you for the quick response.

I did raise a support ticket for this with the vendor, but I haven't had a response yet. I will update the ticket as soon as I get a response

github-actions[bot] commented 5 months ago

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable. It will be closed soon unless the stale label is removed by a committer, or a new comment is made.