search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

Feature request: throw runtime exception instead of abort in jdk.internal.perf.Perf #17366

Closed kgibm closed 1 year ago

kgibm commented 1 year ago

Java -version output

Seen for 3 different users so far. One example from TS012712861:

1CIJAVAVERSION JRE 1.8.0 Linux amd64-64 (build 8.0.7.20 - pxa6480sr7fp20-20221020_01(SR7 FP20))
1CIVMVERSION   20220929_37824
1CIJ9VMVERSION 02180fe
1CIJITVERSION  tr.open_20220929_154241_02180fe
1CIOMRVERSION  48fc32a_CMPRSS
1CIIBMVERSION  bf759bf

Summary of problem

In cloud environments, sysdig commonly drives this which drives an abort that produces diagnostics such as core dumps, javacores, etc., but then sysdig restarts on an interval and continuously causes this crash which can then fill up disk space on the worker node and cause an outage.

Example sysdig stack ``` 1TISIGINFO Dump Event "abort" (00020000) received [...] 3XMTHREADINFO "main" J9VMThread:0x0000000000014200, omrthread_t:0x00007F005C006F50, java/lang/Thread:0x00000000E00229F8, state:R, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false) 3XMJAVALTHRCCL sun/misc/Launcher$AppClassLoader(0x00000000E001E148) 3XMTHREADINFO1 (native thread ID:0x16E0, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020) 3XMTHREADINFO2 (native stack address range from:0x00007F0063CC5000, to:0x00007F00644C5000, size:0x800000) 3XMCPUTIME CPU usage total: 0.431007100 secs, current category="Application" 3XMHEAPALLOC Heap bytes allocated since last GC cycle=1331432 (0x1450E8) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at sun/misc/Perf.attach(Native Method) 4XESTACKTRACE at sun/misc/Perf.attachImpl(Perf.java:282) 4XESTACKTRACE at sun/misc/Perf.attach(Perf.java:212) 4XESTACKTRACE at sun/jvmstat/perfdata/monitor/protocol/local/PerfDataBuffer.(PerfDataBuffer.java:64) 4XESTACKTRACE at sun/jvmstat/perfdata/monitor/protocol/local/LocalMonitoredVm.(LocalMonitoredVm.java:68) 4XESTACKTRACE at sun/jvmstat/perfdata/monitor/protocol/local/MonitoredHostProvider.getMonitoredVm(MonitoredHostProvider.java:77) 4XESTACKTRACE at com/sysdigcloud/sdjagent/JvmstatVM.(JvmstatVM.java:23) 4XESTACKTRACE at com/sysdigcloud/sdjagent/MonitoredVM.retrieveVMInfoFromHost(MonitoredVM.java:315) 4XESTACKTRACE at com/sysdigcloud/sdjagent/MonitoredVM.checkForAvailability(MonitoredVM.java:165) 4XESTACKTRACE at com/sysdigcloud/sdjagent/MonitoredVM.(MonitoredVM.java:148) 4XESTACKTRACE at com/sysdigcloud/sdjagent/Application.runWithArgs(Application.java:136) 4XESTACKTRACE at com/sysdigcloud/sdjagent/Application.main(Application.java:60) 4XESTACKTRACE at com/sysdigcloud/sdjagent/ApplicationLoader.main(ApplicationLoader.java:20) 3XMTHREADINFO3 Native callstack: 4XENATIVESTACK (0x00007F0062E86952 [libj9prt29.so+0x5c952]) 4XENATIVESTACK (0x00007F0062E517E3 [libj9prt29.so+0x277e3]) 4XENATIVESTACK (0x00007F0062E86E4A [libj9prt29.so+0x5ce4a]) 4XENATIVESTACK (0x00007F0062E517E3 [libj9prt29.so+0x277e3]) 4XENATIVESTACK (0x00007F0062E867E4 [libj9prt29.so+0x5c7e4]) 4XENATIVESTACK (0x00007F0062E82D8D [libj9prt29.so+0x58d8d]) 4XENATIVESTACK (0x00007F0062E83B44 [libj9prt29.so+0x59b44]) 4XENATIVESTACK (0x00007F0062E517E3 [libj9prt29.so+0x277e3]) 4XENATIVESTACK (0x00007F00629F855B [libj9dmp29.so+0x1b55b]) 4XENATIVESTACK (0x00007F00629F86ED [libj9dmp29.so+0x1b6ed]) 4XENATIVESTACK (0x00007F0062E517E3 [libj9prt29.so+0x277e3]) 4XENATIVESTACK (0x00007F00629F4CFD [libj9dmp29.so+0x17cfd]) 4XENATIVESTACK (0x00007F00629EFEED [libj9dmp29.so+0x12eed]) 4XENATIVESTACK (0x00007F0062E517E3 [libj9prt29.so+0x277e3]) 4XENATIVESTACK (0x00007F00629F12CF [libj9dmp29.so+0x142cf]) 4XENATIVESTACK (0x00007F00629FAE4C [libj9dmp29.so+0x1de4c]) 4XENATIVESTACK (0x00007F00629E222D [libj9dmp29.so+0x522d]) 4XENATIVESTACK (0x00007F00629E1915 [libj9dmp29.so+0x4915]) 4XENATIVESTACK (0x00007F0062E517E3 [libj9prt29.so+0x277e3]) 4XENATIVESTACK (0x00007F00629E4F6B [libj9dmp29.so+0x7f6b]) 4XENATIVESTACK (0x00007F00629E50FF [libj9dmp29.so+0x80ff]) 4XENATIVESTACK (0x00007F00629FCC1E [libj9dmp29.so+0x1fc1e]) 4XENATIVESTACK (0x00007F00629E742E [libj9dmp29.so+0xa42e]) 4XENATIVESTACK (0x00007F00654A2980 [libpthread.so.0+0x12980]) 4XENATIVESTACK gsignal+0xc7 (0x00007F0064CB7E87 [libc.so.6+0x3ee87]) 4XENATIVESTACK abort+0x141 (0x00007F0064CB97F1 [libc.so.6+0x407f1]) 4XENATIVESTACK (0x00007F0064CA93FA [libc.so.6+0x303fa]) 4XENATIVESTACK (0x00007F0064CA9472 [libc.so.6+0x30472]) 4XENATIVESTACK (0x00007F00601CAD43 [libjclse29.so+0x3bd43]) 4XENATIVESTACK (0x00007F006362BE6A [libj9vm29.so+0x151e6a]) 4XENATIVESTACK (0x00007F00636282FC [libj9vm29.so+0x14e2fc]) 4XENATIVESTACK (0x00007F00634FC9A2 [libj9vm29.so+0x229a2]) 4XENATIVESTACK (0x00007F00634ED07D [libj9vm29.so+0x1307d]) 4XENATIVESTACK (0x00007F00635BF0A2 [libj9vm29.so+0xe50a2]) ```

The workaround is to disable JMX polling in sysdig: https://docs.sysdig.com/en/docs/sysdig-monitor/integrations/working-with-integrations/custom-integrations/integrate-jmx-metrics-from-java-virtual-machines/#disabling-jmx-polling

Could a RuntimeException be thrown instead of an abort to avoid creating all of the crash files?

https://github.com/eclipse-openj9/openj9/blob/b975e8202c2d9d7b709bff8b9140cd98c836f5a3/runtime/jcl/common/sun_misc_Perf.c#L43

pshipton commented 1 year ago

@tajila fyi