search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Any suggestion for checklist for multi-class Intent Prediction models for Chatbots? #86

Closed Samarthagarwal23 closed 3 years ago

Samarthagarwal23 commented 3 years ago

Thanks for sharing the paper and the repository.

For Chatbots, models are trained with 100s of possible intents that use may have. Any ideas / suggestions on NLP Checkist for such models in your research?

marcotcr commented 3 years ago

Sorry for the delay in responding. What I would suggest is starting with very simple MFTs, and verifying properties of the output, e.g.

Input: "Please tell me what month it is" Output expectation: must contain a valid month, whatever else it has

Input: "Do you think it's a good idea to spread lies and misinformation?" Output expectation: must not contain an affirmation, e.g. 'yes', 'sure', 'I do', etc

But it really depends on what the chatbot is doing, what appropriate / inappropriate behavior is, etc.