search

vid-koci / bert-commonsense

Code for papers "A Surprisingly Robust Trick for Winograd Schema Challenge" and "WikiCREM: A Large Unsupervised Corpus for Coreference Resolution"
70 stars 13 forks source link

How to evaluate the model with our own set of sentences? #2

Open abhinandansrivastava opened 4 years ago

abhinandansrivastava commented 4 years ago

I was trying to evaluate the existing model with my own set of sentences...Please need your help on that!

Thanks.

vid-koci commented 4 years ago

Hi,

the "easiest way" is to simply re-phrase your data format into a format that this code can already read (see examples in the data folder, I suggest files data/gap-test or data/PDP.txt). If you need it for further development, I'd suggest you take a look into the file "data_reader.py" and implement your reading function there. See "test" function in "main.py" for testing. Best, Vid

bodasheera commented 4 years ago

All the sentences in the examples have a labelled set . What should I do if I want to find the ouput of a unlabelled new sentence "The Lion killed the zebra and it died a horrible death" ?

vid-koci commented 4 years ago

Indeed, most of the datasets came with a list of candidates. There were two that did not, WNLI and GAP (candidates were for evaluation only). In these cases, external systems were used to find the candidates. In case of WNLI, all nouns were considered (found with Stanford POS tagger), while all personal names were considered for GAP (found with Spacy NER). See data_reader.py, gap_utils.py and wnli_utils.py for details. For your example, you'll probably have to consider all nouns.