Rocketchat chatbot PoC

[ShellyXueHan]

Describe the issue This is to explore options to create a Rocketchat chatbot to interact with community questions.

Background and project goal

Goal 1: There has been many duplicated OCP how-to questions asked in the public channels on Rocketchat. We'd like to leverage a RAG GenAI chatbot, available within Rocketchat, to answer those simple but repetitive questions from the community.

In addition, leverage the chatbot and questions from the community to discover missing/wrong information pieces from our platform knowledge base.

Goal 2: Within BCGov there have been many GenAI chatbot initiatives and project plans. We'll work with EA team to explore the different AI offerings from Public Cloud Platforms to provide in depth information and guidance for the other teams.

Explore Service and integrations

• Generative AI: Google Vertex, AWS Bedrock, Azure OpenAI
• Chatbot integration: custom built chat app that integrates with Rocketchat and the AI service

Evaluation Plan

During the POC period, we'll evaluate the chatbot based on the quality of answers provided by the chatbot. More details here.

After the POC, we will leverage the following for value assessment (more indicators to come as we explore):

• percentage of questions handled by Rocky vs human
• quality of the answer provided by the chatbot
• the time and effort saved by the chatbot vs. the time and effort required to maintain the chatbot

[ShellyXueHan]

Updates:

GCP chatbot released for testing, here are some findings Pros:

• stackoverflow content
• reference link to the actual doc
• ppl like to play with the generative fallback capacity

cons:

• generative fallback quality not reliable, gives incorrect information when it's outside of the BCGov knowledge base (investigating)
• when a topic has been mentioned multiple places in the KB, sometimes it'll give the not-so-relevant content