search

BharatSahAIyak / kisai-bot

0 stars 0 forks source link

QA: Update the tesing suite on CI #525

Open techsavvyash opened 2 months ago

techsavvyash commented 2 months ago

Description

Since deptt UAT is done for RJAI with minimal flags on answers, it is a good time to update the QA testing suite on CI.

This task is required for the following reasons

  1. Testing out and replacing the OpenAI based classifier with BERT classifier
  2. CI has very outdated tests which are not relevant anymore since the PoP book chunks have been updated.
  3. To keep track of things where bot is failing and have a general sense of confidence
PriyanshiSoni11 commented 2 months ago

  1. Generate answers for each of the 750 questions by running a simple script to store and save those answers in a google sheet - Done (Sheets has been shared by @sumitmagico sheet2.csv Sheet1.csv

  2. Update the CI to run on these 750 questions - Executed on local

  3. Update the CI to include tests for telemetry as well - Telemetry dump can not be created due to missing events when making direct query on socket.

  4. Make sure to add telemetry dump of each message in the downloadable CSV report from GitHub actions - Test scripts incudes that but Telemetry event can not be created due to missing events when making direct query on socket.

  5. Reports to be shared in the format as outlined by QA in Golden Dataset sheet. - DOne

PriyanshiSoni11 commented 2 months ago

@Sreejit-K Found a bug for queries which took more than 60 sec to respond, have response same at query Bug-queries.csv

PriyanshiSoni11 commented 2 months ago

Retested, Same queries are responding under 60 sec. Issue is not reproducing.

@Sreejit-K Found a bug for queries which took more than 60 sec to respond, have response same at query Bug-queries.csv