Implement issue classifier to quickly spot the area of improvement on GAIA

[krisnaBukitVista]

DESCRIPTION

This project aims to enhance the error reporting system on GAIA by implementing a detailed issue classifier. The goal is to distinguish between technical and knowledge-based errors, categorize them for better understanding, and visualize the data on DataVista. This improvement will streamline the identification of issues, reduce manual effort in searching for errors, and provide actionable insights for backlog tasks.

PROBLEM

• There is no detailed classifier to label whether an issue is technical or knowledge-based.
• Currently, logs only indicate a generic "is error" without specific details, making it difficult to understand the nature of the problem.
• Manual searching is required to identify the cause of the error, which slows down the process of formulating backlog tasks.

SOLUTIONS

• Enhance the validation process to label each error as either a technical issue or a knowledge issue.
• For technical issues, the system should categorize them into specific types: goal setting issues, validation issues, tool errors, server issues, or database issues.
• Implement visualization of these classified errors on DataVista to measure and analyze them effectively.
• Integrate an intuitive recommendation system that suggests corrective actions based on the type of error detected. For example, if there is a tool error on "get_extra_service," the recommendation should be, "Please fix the tool on get_extra_service."

MEASUREMENT METRICS

• Each event information should include a label indicating whether the error is a technical or knowledge issue.
• The labels should provide detailed classifications, such as goal setting issues, validation issues, tool errors, server issues, or database issues.
• Logs of these detailed errors should be stored in the database.
• The classified errors should be visualized on DataVista, allowing for analysis over time.
• The system should provide intuitive recommendations for each error detected, guiding users on the appropriate corrective actions.

SLA

• Ensure that all error messages include a functioning label indicating whether the issue is technical or knowledge-based.
• The label should include detailed reasoning behind the error.
• Store logs of these detailed errors in the database.
• Provide visualization of the error data on DataVista, presented in a time series format for trend analysis.
• Implement an intuitive recommendation system that provides actionable suggestions based on the type of error detected.

Team Member

@rifqinvnd

[Vidiskiu]

Overall Point: 5.4

Functional Complexity: 1.2

Creating an issue classifier is a substantial enhancement that affects how errors are reported and categorized, which involves intricate logic.

Technical Complexity: 1.3

Developing a system to distinguish and categorize errors based on type and integrate with existing tools like DataVista entails a high technical challenge.

UI/UX Complexity: 0

The issue did not specifically mention UI/UX changes, hence no points awarded for UI/UX complexity.

Data Manipulation: 0.9

Manipulating and storing detailed logs with the new classification system involves complex data handling and database interactions.

Testing: 0.4

Testing the classifier and ensuring the system's outputs are reliable and accurate is key, but it aligns with usual QA best practices.

Dependencies: 0.3

While there might be dependencies such as the integration with DataVista, these are not expected to be highly complex.

Risk and Uncertainty: 0.7

Introducing a new classification system presents risks in terms of accurately identifying and labeling errors, and requires solid validation to ensure correctness.

User Impact: 0.6

The classifier will have a noticeable impact on how users identify and troubleshoot issues, although the impact is moderate as it targets the error reporting process.

[Vidiskiu]

@krisnaBukitVista to check back with ardy on the implementation (include Titan). Then, develop second iteration.

[krisnaBukitVista]

unfortunately, based on several sampling testing, the current system still fail to identify the issue.

• Failed process agent: failed Hermes service categorized as knowledge issue: https://bukitvista.slack.com/archives/C05MR22R31S/p1725956791983669
• Validation error got labelled as tool error: https://bukitvista.slack.com/archives/C05MR22R31S/p1725958770817729
• For tool error, there should be data to mention the specific tool error. We should be able to retrieve the highest tool error within certain period of time