About how to process data

[happypostgraduate123]

I am also learning about AIOps-related content now, and I am very interested in the open source work you provide related to AIOps, especially KGDiag and DiagFusion. However, these projects seem to be missing or unclear about data processing work. I can talk to you. Get in touch? Thanks.

[PerfectZyx]

Thank you for your interest in the work of KGDiag.

In the data processing process, first KGDiag serialises the multimodal data of each microservice instance. Specifically, assuming a one-minute interval for data collection, for metric data, only standardization is required due to the inherent time-series attributes of metric data; for log data, the frequency of occurrence of each type of log template within each window is counted in one-minute window units and sliding steps, and then the logs are converted into continuous time series data; for trace data, the response time, error count, and query count within each window are recorded separately by subdividing the data into For the trace data, KGDiag generates three time series for each pair of call relationships <caller, callee> by subdividing the data into one-minute intervals and recording the three key performance indicators of response time, error count, and query count in each window. KGDiag then uses a 3-sigma method to classify the time series data generated for each minute into three categories: normal, excessively high, and unexpectedly low, and defines the different types of edges in the graph based on these classifications.