grafana / pyroscope-java

pyroscope java integration
Apache License 2.0
83 stars 32 forks source link

Segfault in libasyncProfiler-linux-x64-a38d71b1e75a7f3c6677da15114e73312b6a9e.so+0x1a2cd #35

Closed srt closed 2 years ago

srt commented 2 years ago

Under load I see my JVM segfaulting in libasyncProfiler-linux-x64:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f3d241e72cd, pid=1, tid=24
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.2+8 (17.0.2+8) (build 17.0.2+8)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (17.0.2+8, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libasyncProfiler-linux-x64-a38d71b1e75a7f3c6677da15114e73312b6a9e.so+0x1a2cd]  Profiler::dumpCollapsed(std::ostream&, Arguments&)+0x4bd
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.1)
#
# An error report file with more information is saved as:
# //hs_err_pid1.log

The JVM is run inside a docker container, base image is eclipse-temurin:17 (build 17.0.2+8), pyroscope agent is v0.6.0

srt commented 2 years ago

hs_err_pid1.log

korniltsev commented 2 years ago

I reproduced segfault # C [libasyncProfiler-linux-x64.so+0x1a2cd] Profiler::dumpCollapsed(std::ostream&, Arguments&)+0x4bd Repro:

``` import one.profiler.AsyncProfiler; import one.profiler.Counter; import one.profiler.Events; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; public class App { public static void main(String[] args) { int n = 16; ExecutorService executors = Executors.newFixedThreadPool(n+1); executors.submit(new ProfilerLoop()); for (int i = 0; i < n; i++) { executors.submit((Runnable) () -> { while (true) { fib(32L); } }); } } private static Long fib(Long n) { if (n == 0L) { return 0L; } if (n == 1L) { return 1L; } return fib(n - 1) + fib(n - 2); } private static class ProfilerLoop implements Runnable { @Override public void run() { // AsyncProfiler instance = AsyncProfiler.getInstance("/mnt/hgfs/share/libasyncProfiler-linux-x64.so"); AsyncProfiler instance = AsyncProfiler.getInstance("/mnt/hgfs/share/async-profiler/libasyncProfiler.2.8-ea.so"); while (true) { instance.start(Events.CPU, 10_000_0); try { Thread.sleep(10); } catch (InterruptedException e) { Thread.currentThread().interrupt(); break; } System.out.println("."); for (int i = 0; i < 20; i++) { instance.dumpCollapsed(Counter.SAMPLES); } instance.stop(); } } } } ```

It does not segfault when upgraded to async-profiler 2.8-ea It is fixed here https://github.com/jvm-profiling-tools/async-profiler/commit/fe173c41019e443ef23ffd886b5df9f147e0ece3 We should wait for a stable 2.8 release and upgrade.

korniltsev commented 2 years ago

@srt We've updated async-profiler to 2.8. The crash should de gone. Please reopen if it happens again.