krisnaBukitVista commented 3 weeks ago

DESCRIPTION

We aim for Thea to adopt key communication principles and enhance her ability to detect human emotions, making her not just effective in solving cases but also more likable. By providing emotional closure to partners, she can build stronger connections beyond just delivering answers.

PROBLEM

Currently, Thea focuses on providing direct answers, but sometimes an immediate answer isn’t what’s needed. Offering closure is essential to ensure the partner is ready to accept the information.
Thea should deepen her understanding of communication theory to behave in a more natural, human-like manner.

SOLUTIONS

Ensure Thea understands communication principles, including active listening, reciprocity, politeness theory, emotional contagion, social penetration, conversational mimicry, personalization, and context awareness.
Enable Thea to accurately label emotions and respond accordingly.
The full list of communication principles and sentiment handling can be viewed here

MEASUREMENT METRICS

Thea should be capable in adopting mirroring skill
Thea should accurately label emotions with a high degree of correctness.
Thea should respond appropriately based on the detected emotions.
Thea should consistently apply the set communication principles.

SLA

Thea should adopted mirroring skill with at least 75% quality answer based on supervisor evaluation
Thea should label emotions with at least 75% accuracy.
Thea should handle the detected emotions with 75% accuracy in terms of appropriate responses.
The violation rate for communication principles should not exceed 20%.

EVALUATION RESULT

Labelling Skill

Current Status: After reviewing 20 sample conversations, we observed some inconsistencies in Thea's labelling skill. Out of the 20 samples, incorrect labels were assigned to 3 conversations, resulting in a reliability rate of 85%.
- Key Issues Identified: There are three emotions that Thea tends to confuse:
- Curiosity or Interest: Sometimes mislabeled as neutral or disinterest.
- Concern or Anxiety: Misinterpreted as confusion or general inquiries.
- Skepticism or Doubt: Often confused with concern or frustration.
Suggested Fix: Adjusting the emotion descriptions and enhancing Thea’s ability to differentiate subtle emotional cues would help improve labelling accuracy.

Mirroring Skill

Current Status: The mirroring skill has shown significant issues. Out of 10 samples, 5 were unsatisfactory, giving us a reliability rate of 50%.
Key Issues Identified:
- Exact Copying: In some cases, Thea mirrors the guest’s message without paraphrasing or adding value, leading to a simple copy-paste effect.
- Incorrect Focus: Thea sometimes mirrors the wrong part of the sentence, focusing on less relevant details instead of the main idea.
Suggested Fix: To improve this skill, Thea should be trained to:
- Paraphrase more effectively rather than repeating verbatim.
- Focus on key ideas within the guest’s message to ensure the response is contextually relevant and useful.

Overall Performance

Categorization Accuracy: Thea performs well in categorizing topics accurately, with no major issues in this area. However, there is a growing need for more specific categories based on conversations.
Request for New Categories: From the analysis of 20 samples, 5 conversations highlighted the need for new, dedicated categories. This suggests that the current categories are not fully capturing the range of discussions taking place.
Technical Issues:
- Bugs Identified: Two technical bugs were found in the 20 samples. These need to be addressed for smoother operation.
- Conversation Display Issues: Users are currently unable to view full conversations, which is a critical issue that affects overall transparency and review processes.

Summary of Actionable Points

Emotion Descriptions Update: Tweak the descriptions for curiosity, concern, and skepticism to reduce confusion in labelling.
Mirroring Skill Improvement: Train Thea to paraphrase more effectively and focus on the core idea of the conversation.
New Categories: Collaborate with Project-GAIA to create new categories based on identified gaps in the current structure.
Technical Fixes: Address the bugs and improve the "show full conversation" functionality for better user experience.

Team Member

@Ardy-BukitVista @rifqinvnd

Vidiskiu commented 1 week ago

Overall Point: 5.4

Functional Complexity: 1

Enhancing communication principles for AI and enabling emotion detection are advanced functionalities that involve complex processing and understanding of communication theories.

Technical Complexity: 1.1

This implementation requires sophisticated technical algorithms for natural language processing and emotion recognition, pushing towards the upper end of technical complexity.

UI/UX Complexity: 0

The issue description does not specify UI/UX changes, so no points are awarded.

Data Manipulation: 0.9

High level of data manipulation is needed for processing and interpreting language, emotions, and implementing communication theories within the system.

Testing: 0.5

Rigorous testing will be required to ensure the features work as intended and that the AI's emotion detection and responses are accurate.

Dependencies: 0.4

The project might rely on external NLP and AI libraries or frameworks, though these dependencies are not directly specified.

Risk and Uncertainty: 0.5

Incorporating such a sophisticated set of features into Thea involves a significant risk of incorrect implementation, which could impact user experience.

User Impact: 1

Improving Thea's communication and emotion detection capabilities will have a high impact on user engagement and satisfaction.