search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

NER tests #41

Closed elliotbart closed 4 years ago

elliotbart commented 4 years ago

Hi, First thanks for your great work and this very useful library.

I am looking to test NER models (Transformer and LSTM based), also I would like to know if you had any example/code of how you could test such models? I haven't found any, even in the notebook 5. Testing transformer pipelines.

I guess the key is to be able to make an expectation function at token-level ? Maybe you already explored something ?

Many thanks!

marcotcr commented 4 years ago

I should write some expectation helper functions to make this easy, but I have to play with this a little bit to figure out what makes sense. In the meantime, here is a notebook with an example of a token-level expectation function.

Please let me know if there are any expectations that are hard for you to express, that would be really helpful.

elliotbart commented 4 years ago

Exactly what I was looking for as a starter. Many thanks !