Hello, thank you for your interest. Train data, too, is processed in a way so that the first candidate is always the correct one. Note that the model is agnostic to the fact whether a candidate is the first or the second candidate, it simply assigns the current candidate a score without access to the other candidate (or the knowledge whether it's dealing with the first or the second candidate). The order in which the candidates are processed thus does not make any difference.

vid-koci / bert-commonsense

Code for papers "A Surprisingly Robust Trick for Winograd Schema Challenge" and "WikiCREM: A Large Unsupervised Corpus for Coreference Resolution"

70 stars 13 forks source link

Hello, thank you for your interest. Train data, too, is processed in a way so that the first candidate is always the correct one. Note that the model is agnostic to the fact whether a candidate is the first or the second candidate, it simply assigns the current candidate a score without access to the other candidate (or the knowledge whether it's dealing with the first or the second candidate). The order in which the candidates are processed thus does not make any difference. #4

Closed jc-ryan closed 3 years ago

jc-ryan commented 3 years ago

Hello, thank you for your interest. Train data, too, is processed in a way so that the first candidate is always the correct one. Note that the model is agnostic to the fact whether a candidate is the first or the second candidate, it simply assigns the current candidate a score without access to the other candidate (or the knowledge whether it's dealing with the first or the second candidate). The order in which the candidates are processed thus does not make any difference.

Hope that answers your question!

Originally posted by @vid-koci in https://github.com/vid-koci/bert-commonsense/issues/3#issuecomment-799326718

jc-ryan commented 3 years ago

Thanks for your reply, but the margin loss always takes loss1 as the loss of correct one, how about the condition when the candidate_2 is correct ?

vid-koci commented 3 years ago

Hello, if you check the data_reader.py file, you'll see that data is always read in a way that the first candidate is the correct one. When the second one is correct, they are simply swapped.