question about suite.to_raw_file

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

MIT License

2.01k stars 204 forks source link

question about suite.to_raw_file #64

Closed ggxxding closed 3 years ago

ggxxding commented 3 years ago

Hi, I see that I can generate my own suite with : suite.to_raw_file('/tmp/temp', n=500, seed=1) path = '/home/marcotcr/work/checklist/release_data/sentiment/sentiment_suite.pkl' suite.save(path)

Then I can run other model to predict the suite and run .pkl file to test it. But the order of words and sentences in '/tmp/temp' seems to be random when I change n and seed in the code. When I use .pkl file to test my model, will the random order of the test cases affect the testing?

marcotcr commented 3 years ago

Every time you run suite.to_raw_file, it keeps track of which examples were randomly chosen and the order. When you run suite.run_from_file, it assigns each instance to the appropriate test case.