marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Synonyms function fails if templates arg is a list or tuple #77

Closed ramji-c closed 3 years ago

ramji-c commented 3 years ago

editor.synonyms() calls editor._wordnet_stuff() which in-turn invokes editor.template() before checking if the arg word is present all the strings passed in templates arg. However, template() function expects an unroll argument to be True in order for a templates list to be properly unrolled. However, synonyms() doesn't check the type of templates arg passed to it and subsequently doesn't pass-along an unroll arg to the template() function. This results in the following error

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-9-e7949edd59ad> in <module>
      3                            "how do I change my car type"],
      4                 word="change",
----> 5                 threshold=10)

~/repos/checklist/checklist/editor.py in synonyms(self, templates, word, threshold, **kwargs)
    395         if type(templates) in [list, tuple]:
    396             unroll = False
--> 397         return self._wordnet_stuff(templates, word, 'synonyms', threshold=threshold, unroll=unroll, **kwargs)
    398 
    399     def related_words(self, templates, word, threshold=5, **kwargs):

~/repos/checklist/checklist/editor.py in _wordnet_stuff(self, templates, word, type, threshold, depth, pos, **kwargs)
    345         texts = [texts[i] for i in idxs]
    346         if type != 'related' and any([word not in x for x in texts]):
--> 347             raise Exception('word %s must be in all templates' % word)
    348         fn = {'antonyms': self.tg.antonyms,
    349          'synonyms': self.tg.synonyms,

Exception: word change must be in all templates

as the list of templates is wrapped inside another list, causing the check in line no. 347 to fail.

Steps to reproduce: I wrote this small snippet that recreates this error

from checklist.editor import Editor
editor = Editor()
editor.synonyms(templates=["I would like to change my check-in date",
                                                  "is it possible to change my seat",
                                                  "how do I change my car type"], 
                             word="change",  
                             threshold=10)

A simple fix would be check the templates arg type in either synonyms() or _wordnet_stuff() and automatically set unroll=True if type is list or tuple. I have a working change and can submit a PR if desired.

marcotcr commented 3 years ago

Thanks for investigating the cause of the bug and even proposing a fix. As I understand it, we can just set unroll=True regardless of the templates type, so I did that in 3fff9ea. Please let me know if you think this breaks anything.

marcotcr commented 3 years ago

We check the data type inside editor.template, and only unroll if it's in [list, np.array, np.ndarray, tuple]

ramji-c commented 3 years ago

Thank you @marcotcr for a quick fix. I tested it and is working fine 👍