marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Labels cannot be list of int #49

Closed akobre01 closed 3 years ago

akobre01 commented 3 years ago

The following code gives me an error:

from checklist.editor import Editor
editor = Editor()

exs = ['hello', 'good bye']
labels = [0, 1]
ret = editor.template('{ex}', ex=exs, labels=labels, save=True, meta=True)

Based on the documentation I expect labels to accept a list of ints.

marcotcr commented 3 years ago
    labels : int or object with strings on leaves

labels here expects either a single integer (for the whole template) or an object with strings on the leaves. This is because we allow for templates in labels as well, e.g. you could have labels={'part1': 'not {ex}', 'part2': 'dummy'}

What could be confusing here is that the labels argument in MFT or other test types does accept a list of integers, strings, and etc. Sorry about that, it is confusing : |