search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Feature request Static HTML output #53

Closed timothyjlaurent closed 3 years ago

timothyjlaurent commented 3 years ago

Is it possible to output checklist results in some structured format, eg html, xml.

We'd like to to integrate Checklist as a CI step and want to be able to persist the model performance.

marcotcr commented 3 years ago

If you run test.get_stats(), you get a dictionary with various statistics and test results. If you have a suite, you can run

for test in suite.tests:
    stats = test.get_stats()
    # do something with the stats here, generate html or xml

Does this solve your problem or did I misunderstand what you're looking for?

timothyjlaurent commented 3 years ago

Yes, it does. Thanks for the pointer.

prrao87 commented 3 years ago

Just in case anybody else stumbles across this, what worked for me is the following (suite.tests returns an OrderedDict) :

for test_name, test in suite.tests.items():
    print("test_name")
    stats = test.get_stats()
    # do something with the stats here, generate html or xml