sofastack / sofa-rpc

SOFARPC is a high-performance, high-extensibility, production-level Java RPC framework.
https://www.sofastack.tech/sofa-rpc/docs/Home
Apache License 2.0
3.81k stars 1.17k forks source link

Monitoring rpc call process with Java Flight Recorder #1128

Open EvenLjj opened 2 years ago

EvenLjj commented 2 years ago

Java Flight Recorder (JFR) is a tool used to collect runtime diagnostic data of Java applications. It is integrated in the JDK and collects runtime details of JVM and Java applications through JFR Event. The overall goal: to bury points in each stage of the RPC call process through JFR, and analyze the time-consuming analysis of each stage based on the collected bury point data, so that users can troubleshoot online call timeout issues. Specifically, the following three items need to be completed:

  1. Implement Profile extension points. The implementation of this extension point can be dynamically loaded through the SPI mechanism. The content of this extension point can be regarded as providing callback functions for each stage in the calling process similar to the Tracer module. Different Profile implementations can implement different recording methods through these callbacks.
  2. Provide profile extension point implementation based on JFR. Determine the current RPC call phase by monitoring the SOFA RPC internal event bus, define the corresponding JFR Event at the beginning of the phase, fill in the Event-related metadata in the phase, and submit the Event at the end of the phase.
  3. Provide tools for displaying buried point data. By reading and parsing the JFR event record file, the time-consuming situation of each stage in the calling process is restored, and the related buried points are connected through technical means.

    Java Flight Recorder (JFR)是一个用于收集有Java应用程序运行时诊断数据的工具。它集成在JDK中, 通过JFR Event来收集JVM和 Java应用程序的运行时细节。 总体目标:通过JFR在RPC调用流程中的各个阶段进行埋点,根据收集到的埋点数据对各个阶段进行耗时分析,方便使用者排查线上调用超时问题。具体来说,需要完成以下三个内容:

  4. 实现Profile扩展点。该扩展点的实现可以通过SPI机制进行动态加载。该扩展点的内容可以看作就是对类似Tracer模块中那样提供对调用流程中的各个阶段的回调函数。不同的Profile实现可以通过这些回调实现不同的记录方式。
  5. 提供基于JFR的Profile扩展点实现。通过监听SOFA RPC内部事件总线判断当前RPC调用阶段,在阶段的开始时刻定义相应的JFR Event,在阶段中对Event相关的元数据进行填充,在阶段结束时进行Event的提交。
  6. 提供埋点数据展示工具。通过读取解析JFR事件记录文件还原调用流程中各个阶段的耗时情况,并通过技术手段将相关联的埋点联系起来。
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

pmupkin commented 1 year ago

/assign