krisnaBukitVista commented 1 month ago

Description

This project aims to improve the reliability and consistency of knowledge retrieval and service response within our hospitality platform. The focus is on ensuring accurate data delivery for key functions such as retrieving check-in information, extra services, and responding to specific user inquiries. By addressing these issues, we aim to enhance user satisfaction, reduce response time, and streamline internal processes.

Problem

Inconsistent data retrieval, delayed responses, and unreliable information sharing have led to issues in user experience. Users often face long wait times for critical information such as keybox codes, listing links, and extra services, which can result in dissatisfaction and operational inefficiencies. Additionally, there is a need to improve the system’s ability to generate reviews and respond reliably to driver service requests.

Feedback Issues:

There is a possibility that the Chroma it is connected to might not be the new version of Chroma, potentially leading to outdated responses.
The tools integrated for the employee user type may need an evaluation to see if they can be merged with more updated tools or improved within the existing system.

Solution

Ensure Knowledge Retrieval from Collection hospitality_experience: We will improve the system's ability to access and retrieve necessary data from the hospitality experience collection to ensure that user receive timely and accurate information.
Ensure Reliable Keybox Code Delivery: Strengthening the system to consistently provide the correct keybox codes when requested will improve the check-in process for user.
Provide Reliable Listing Link When Requested: We will enhance the functionality to return accurate listing links, improving communication and ensuring that user always receive the right information about their bookings.
Consistent Data Storing for Extra Service Between Chroma and Vendor List: Streamlining data storage processes will reduce discrepancies between Chroma and the vendor list, ensuring a smooth experience for user requesting extra services.
Improve Data Retrieval for Check-In Policy: Enhancing the system’s ability to pull up-to-date check-in policy information will reduce errors and miscommunication around user arrival procedures.
Add Tools to Generate a Review List Based on Conditions: Implementing tools that can generate lists of reviews based on specific conditions will help partners and internal teams analyze feedback efficiently and respond to trends.
Eliminate Server Delays: Optimizing server performance will prevent delays in data retrieval and ensure faster response times for all user and partner inquiries.
Increase Reliability in Responding to Driver Service Requests: Improving the system’s capacity to address driver service inquiries accurately will minimize user confusion and reduce the need for manual intervention.
Upgrade to the New Version of Chroma: Investigate whether the Chroma system connected to the "ask-anything" feature is outdated and, if so, update it to the latest version.
Evaluate and Update Tools for Employee Users: Review the tools available to the employee user type and determine if they can be merged with more updated tools or if the existing ones need internal updates.

Measurement Metrics

Percentage of user requests related to the topics above, including keybox codes, listing links, extra services, check-in policies, reviews, and driver services.
Tracking the response accuracy and consistency for each of these requests.

SLA

Ensure that at least 90% of messages on the relevant topics are responded to correctly within the first week of evaluation.

Evaluation

Mid Evaluation (Nov 14th)

After testing on 20 samples with 7 topics, we achieved 14 correct answers and encountered 6 incorrect responses, resulting in a reliability rate of 70%.

This outcome falls below our SLA target. To address this gap, I will create an additional backlog item to focus on enhancing accuracy and reliability.

Observation: Three topics show particularly low success rates:

Retrieving collections on hospitality_experience
Listing link retrieval
Requests related to drivers

For reference, the full sample dataset and test results are documented in the following link: here

Final Evaluation

Objective: To assess GAIA's performance across three key abilities using 15 test samples.

Focus Areas:

Retrieving collections on hospitality_experience
Listing link retrieval
Requests related to drivers

Results: GAIA was tested using 15 samples, with an even distribution across the three focus areas. The system demonstrated a 100% success rate in accurately answering all queries in each category.

Conclusion: Based on the results, GAIA met the SLA requirements and performed flawlessly across all tested topics. This confirms the project's success and readiness for deployment or further integration.

krisnaBukitVista commented 4 days ago

Sample: Hospitality Experience Retrieval

Link Retrieval

Driver-Related Requests

Vidiskiu commented 4 days ago

Overall Point: 6.25

Functional Complexity: 1.5

The feature involves enhancing several aspects of knowledge retrieval which includes accessing collections, generating specific lists, and responding to different types of requests which adds moderate functional complexity.

Technical Complexity: 2

This enhancement encompasses improvements in data storage, server performance optimization, and possibly upgrading or integrating systems such as Chroma, which indicates a high technical complexity.

UI/UX Complexity and Impact: 0.75

Although the issue focuses more on backend processes, ensuring that the information is consistently and reliably presented requires some UX consideration, particularly in the context of presenting feedback, listing links, and error handling.

Testing and Quality Assurance: 1

The issue requires thorough testing to ensure data accuracy and server reliability, necessitating various tests like integration, regression, and performance tests.

Risk and Dependencies: 1

The issue carries a high risk due to its impact on user satisfaction and operational efficiency. Dependencies on external software updates and integrating various tools also add to the risk.

Bukit-Vista / roadmap

Enhancing Knowledge Retrieval and Response Accuracy in Ask-anything #75