search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Is there a way to save all the failed test cases in some text file or in a list? #125

Closed nishnk-vishi closed 2 years ago

nishnk-vishi commented 2 years ago

We need to save the all the failed test cases in a text file. Is there a way to save them??

marcotcr commented 2 years ago

Do you only want the failed ones? If you want all tests, there is test.to_raw_file('/tmp/raw_file.txt') (see lines 16-21 of this).

If you only want the failed ones, you can do the same thing (but you need to call test.to_raw_file before running it, because that messes up the index), and get the index of failed testcases from test.results.passed (where True means it passed the test, so you just filter by False)