search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Test suite example notebook #104

Closed nking-1 closed 3 years ago

nking-1 commented 3 years ago

This notebook explains how to create a test suite, as well as how to read the model's predictions from a file.

View the notebook here: https://github.com/Nking92/checklist/blob/test-suite-notebook/notebooks/Test%20Suite%20Examples.ipynb

I also found a small error in the MFT examples notebook, so I included that fix here as well. This expectation function wasn't implemented correctly:

def response_contains_correct_language(x, pred, conf, label=None, meta=None):
    country = meta['country']
    language = correct_responses[country]
    return language in pred