i-dot-ai / caddy-chatbot

Caddy is an AI powered co-pilot for customer service functions everywhere.
https://ai.gov.uk/projects/caddy/
MIT License
15 stars 4 forks source link

Mh eval #151

Open mhgov opened 3 months ago

mhgov commented 3 months ago

Evaluation notebooks

Still has some outstanding actions for Alex/Andreas to pick up when required (bold)

For topic modelling (Alex) PS: We want to understand the variation in approval rate for different topics so we can focus are efforts to add content, revise script on specific Steps: 1) Locate data of historical messages / responses in download from s3 (done) 2) Use topic modelling to categorise the message / responses (done) 3) Look at the approval rate of the messages (done) 4) Use the router instead of topic modelling and recreate same analysis (to do)

For rct evaluation (Andreas) PS: Need to prep a dataset and a notebook in case of an FOI request Steps: