search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

NER example #25

Closed ninamalina closed 4 years ago

ninamalina commented 4 years ago

Hi! In the paper you describe some invariant tests for NER models, however, in the library I can't find any code to this. Could you share this as well? Thanks :)

tongshuangwu commented 4 years ago

Hi! I don't think our papers involved tests for [NER models] though; Did you mean the invariance tests w.r.t [capability: named entity] for sentiment/QQP/squad models? It was a capability we tested, e.g., changing the named entity from Dallas -> Seattle should not change the sentiment in "I liked my flight to Dallas." If so, those tests are in the paper testing notebooks

Please feel free to reopen if this doesn't answer your question. I can try to further clarify if you can point me to which tests you are referring to in the paper :)