[Feature] Support Frontend Log Aggregation

[dohard-ma]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

While SkyWalking already provides powerful log collection and querying capabilities, there are still some limitations in terms of frontend log aggregation. We would like to propose adding support for frontend log aggregation in SkyWalking, allowing for better monitoring and analysis of log data generated by frontend applications. This enhancement would contribute to improved efficiency in troubleshooting and enhancing user experience.

Use case

1. Enable frontend applications to report log data to SkyWalking and support aggregating multiple log messages into a single entry.
2. Provide flexible configuration options for setting rules and conditions for log aggregation.
3. Introduce a visual log aggregation analysis tool within the SkyWalking interface, enabling users to quickly view and analyze aggregated frontend log results.
4. Support exporting aggregated log data for further in-depth analysis and archiving.

By implementing this feature, we believe that SkyWalking will be better equipped to meet the monitoring and analysis needs of frontend applications, leading to enhanced efficiency in issue diagnosis and overall user satisfaction. We look forward to the consideration and implementation of this feature. Thank you.

Related issues

No response

Are you willing to submit a PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wu-sheng]

Are you going to do this? And is this through client-js lib?

[wu-sheng]

Provide flexible configuration options for setting rules and conditions for log aggregation.

Could you explain what log aggregation rule is about? The LAL is designed for log process and aggregation(metric extractor). What is new here?

[dohard-ma]

The skywalking-client-js is responsible solely for reporting error logs, while the aggregation, counting, and other operations on the reported log data are reliant on the backend.

Below are examples of functionalities provided by Sentry:

[wu-sheng]

How do you aggregate? We don't recommend postponed aggregation as it is super slow. Could you demo how could you count users in current OAP architecture?

[wu-sheng]

@MaShizhen Any update? You open this issue for 5 days, but no further update.

[dohard-ma]

hi @wu-sheng. Thank you very much for your guidance and reply. My expectation for the log panel is to be able to achieve more detailed aggregation functions, and to be able to click on the record to view error information including stacks, etc., rather than simply counting the occurrence of the same error and displaying it as a metric on the dashboard . According to your valuable suggestions, I carefully read the relevant documents, and have a deeper understanding of the capabilities of LAL (Language Analysis Language). However, I've noticed that LAL may not meet the requirements of my more complex needs.

Based on this, am I understanding correctly? I may need to consider introducing other professional log analysis platforms to achieve the functions I need? Hope to get your further suggestions and guidance, thank you.

[wu-sheng]

I can't suggest anything until you share what is LAL missing, and what you want to add. More importantly, why a new platform? If it is another platform, shouldn't it be a separate/independent project? What is Skywalking role in your design.

After all, I don't have a whole picture of your proposal.

[dohard-ma]

Sorry, I added the full description as below.

Background: I'm currently working on implementing front-end error log reporting. Our selection criteria prioritized solutions that support private deployments and high availability. After careful consideration, we and our back-end colleagues decided to choose SkyWalking. It has proven to be excellent at both, and supports requested link tracing, which is a good match for our requirements. Currently, we have successfully integrated the front-end error log reporting with SkyWalking. However, we found some areas that appeared to lack function, as described below:

Issue 1: Lack of Aggregation Support

Scenario: Frontend error logs can accumulate significantly, especially considering our daily user activity of 50,000 users. On average, each user generates one error log per day. Consequently, this results in approximately 50,000 error logs daily. Displaying every individual error log in the log panel would lead to an extensive list, making it challenging to navigate and analyze. To address this, we seek the ability to aggregate error logs. For instance, displaying only a single instance of the same error log per day would significantly reduce the length of the log panel's listing, enhancing ease of use.

Example of current situation： the result i want

Issue 2: The front-end log panel does not support search

Scenario: When a user reports that a white screen occurs when using the app, we need to search the user's log through the log panel to locate the problem. We may discuss in this related issue post https://github.com/apache/skywalking/issues/3524

[wu-sheng]

Issue 1: Lack of Aggregation Support

I don't think aggregating based on your way is acceptable for SkyWalking. We are monitoring system performance, rather than doing users' behaviors analysis. So, if you want a pattern match and aggregate to get a number(counter), go ahead. MAL has supported it. But you are proposing the USER ID oriented aggregation, this is clearly a business analysis system, sometimes called BPM(business performance management).

SkyWalking could collect this kind of logs, but doing this kind of aggregation is post-aggregation, and usually run in the bulk mode. This doesn't fit SkyWalking scope.

Issue 2: The front-end log panel does not support search

All content-based searching will be closed most likely in 2024, when we are choosing BanyanDB as the primary storage option. In today's industry practice, the tag could be analyzed and labeled through LAL, and query should base on that.