Issues in KMAI bot - Master Ticket

[Gautam-Rajeev]

Issue Identification

Preparation

<Why KMAI?>

• [x] BharatSahAIyak/kisai-bot#233
• [x] BharatSahAIyak/kisai-bot#234
• [ ] ~Adding prompt to not answering through GPT in case it is not relevant~ @Amruth-Vamshi (Deprioritized)

Create a representative sample of queries for all products

• Create a process to evaluate efficacy/accuracy of every step - Efficacy Params (ETA - 30-06-2024)
• Output - <[Link Sheet]>

Creating a Golden Dataset (Manual)

• [ ] BharatSahAIyak/kisai-bot#235

Automating the testing

@GautamR-Samagra to carry out using collab linked here and to be added to telemetry later

• [x] Using GPT to check if translation is good
• [x] Using GPT to check for spelling mistakes
• [x] Using GPT to check if ASR worked
• [x] Using GPT to check if retrieved chunks had the answer

Identification of Issues

• [x] BharatSahAIyak/kisai-bot#239
• [x] BharatSahAIyak/kisai-bot#304

What all blocks can affect the answer quality -

Resolution

Create test bots that are a copy of KMAI bot and does a combinations of strategies in order of priority of features

Phase 1 (July End) | Scope Increased | New Deadline is 9th August | 2416

• [ ] Quality Issues
  • [x] | QA | Translation Issues in Hinglish - BharatSahAIyak/kisai-bot#306
  • [x] | To be Picked, ETA Thu | BharatSahAIyak/kisai-bot#309
  • [x] | To be Picked, ETA Thu | BharatSahAIyak/kisai-bot#310
  • [x] | ETA - Friday | Translation Issues in Model - https://github.com/BharatSahAIyak/issues/issues/124
  • [x] | To be Picked, ETA Friday | GPT Answering Questions - BharatSahAIyak/kisai-bot#307
  • [x] Retrieval Improvement - Add reranker - https://github.com/BharatSahAIyak/ai-tools/issues/128
  • [ ] | Product Scoping, Data Collection, ETA 9th | Improving Search through tagging - BharatSahAIyak/kisai-bot#232 (moved to phase 1)
    • [ ] Rechunking docs
  • [x] | In Progress | Classifier RLHF Pipeline - https://github.com/BharatSahAIyak/kisai-bot/issues/331 (@GautamR-Samagra to link ticket)
    • [ ] Classification (Agri/Non Agri) Issues - BharatSahAIyak/kisai-bot#305
    • [ ] Classification leading to wrong prompt - BharatSahAIyak/kisai-bot#308
• [ ] Latency Issues
  • [x] | To be Picked, already on RJAI, ETA - Friday | Introduce BGE Embedding - BharatSahAIyak/kisai-bot#267

Phase 2 (August End) | 2417, 2418

• [x] Translation Issues - Post Processing Dictionary - https://github.com/BharatSahAIyak/issues/issues/125
• [ ] BharatSahAIyak/kisai-bot#311
• [ ] BharatSahAIyak/kisai-bot#312
• [ ] BharatSahAIyak/kisai-bot#313
• [ ] Training ASR models (Whisper)
• [ ] BharatSahAIyak/kisai-bot#314
• [ ] Process
  • [ ] BharatSahAIyak/kisai-bot#315

Update History to indicate new telemetry added

• [ ] Check how many questions are actually using trained content by picking top 20 for all questions @GautamR-Samagra
• [ ] BharatSahAIyak/kisai-bot#394

Moving to dev:

• [ ] Move dataset and flow to stage

Next steps for fixing :

• [ ] #447
• [ ] BGE filter fixes
• [ ] #448
• [ ] #449

[ChakshuGautam]

Bot combinations identified

Build bots with combination of all these features and on the outputs of these bots, check which performs best on test set

• [ ] 1. Add tags to all chunks
• [ ] 2. Removing classifier based prompting completely and do plain content search on current tags
• [ ] 3. Using reranker - take top 5 based on classifier chunks, top 5 based on word matching, top 5 from NER, top 6 from chunks and rerank all using reranker
• [ ] Do 3 + self hosted translate
• [ ] Do 3 + dict correction on ngram
• [ ] Do 2 + self hosted translate
• [ ] Do 2 + dict correction on ngram
• [ ] Do 3 with propositions
• [ ] Do 3 with very small chunks

[Gautam-Rajeev]

• [x] Assign someone to improve classifier accuracy from 90% to 97+
• [ ] Link bge or filter ticket here
• [ ] Tune weights for approach 1 transalte - Kartik