(EAI-190): Fix and improve qualitative testing framework - Githubissues

mongodb / chatbot

MongoDB Chatbot Framework. Powered by MongoDB and Atlas Vector Search.

https://mongodb.github.io/chatbot/

Apache License 2.0

106 stars 48 forks source link

(EAI-190): Fix and improve qualitative testing framework #294

Closed mongodben closed 5 months ago

mongodben commented 5 months ago

Jira: https://jira.mongodb.org/browse/EAI-190

Changes

Fix bugs introduced to qualitative testing framework
Refactor qualitative test assertion with the following changes:
- Use OpenAI function instead of Typechat. Done b/c this gives us more control into behavior + Typechat doesn't work very well with ChatGPT 0610 model version.
- Have it do few-shot chain-of-thought prompting for improved accuracy.
Fix broken AVS filter (was using old pre-GA syntax)

Notes

-