grafana / pyroscope-java

pyroscope java integration
Apache License 2.0
83 stars 32 forks source link

Pyroscope Agent core dump #74

Closed vanhtuan0409 closed 1 year ago

vanhtuan0409 commented 1 year ago

I was running Pyroscope agent from within scala code as follow

          val config = new PyroscopeConfig.Builder()
            .setApplicationName("gondor")
            .setProfilingEvent(EventType.ITIMER)
            .setFormat(Format.JFR)
            .setServerAddress("http://x.y.z.n:4040")
            .build()
          PyroscopeAgent.start(config)

The core dump is as follow

gondorAppServer [WARN] Install JVM debug symbols to improve profile accuracy
gondorAppServer #
gondorAppServer # A fatal error has been detected by the Java Runtime Environment:
gondorAppServer #
gondorAppServer #  SIGSEGV (0xb) at pc=0x00007f7d86e62aed, pid=104616, tid=105778
gondorAppServer #
gondorAppServer # JRE version: OpenJDK Runtime Environment (19.0.1+10) (build 19.0.1+10)
gondorAppServer # Java VM: OpenJDK 64-Bit Server VM (19.0.1+10, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
gondorAppServer # Problematic frame:
gondorAppServer # V  [libjvm.so+0xe62aed]  JVM_handle_linux_signal+0x14
gondorAppServer #
gondorAppServer[ERROR] 2022-12-22 22:04:27.583 [INFO] Profiling started
gondorAppServer # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/forked-stargazer/apps/gondor/gondorAppServer/core.104616)
gondorAppServer #
gondorAppServer # An error report file with more information is saved as:
gondorAppServer # /home/forked-stargazer/apps/gondor/gondorAppServer/hs_err_pid104616.log
gondorAppServer [thread 105779 also had an error]

Environment:

openjdk 19.0.1 2022-10-18
OpenJDK Runtime Environment (build 19.0.1+10)
OpenJDK 64-Bit Server VM (build 19.0.1+10, mixed mode)

Pyroscope java SDK version:  0.10.3 (run in native OS host - glibc -arch linux)
Pyroscope server version: 0.36.0 (run within docker)
korniltsev commented 1 year ago

Can you check if latest async-profiler works there? Maybe we just need to update it in our agent.

vanhtuan0409 commented 1 year ago

May I know how to use the latest async profiler

korniltsev commented 1 year ago

'$ ./profiler.sh -d 30 '

You can get more info in async-profiler's readme

vanhtuan0409 commented 1 year ago

seems like a problem of async profiler itself. Using the latest async profiler binary I still got coredump. Tried with JDK17 also no luck

korniltsev commented 1 year ago

Can you share a coredump file itself, running on any java app with no sensitive data in it (hello world)? It should have been dumped to a core.* file You may need to enable it: https://stackoverflow.com/a/17994

korniltsev commented 1 year ago

And maybe hs_err_pid*.log as well

korniltsev commented 1 year ago

I will close the issue for now as of inactivity. Feel free to reopen if you have more bits to share.