how do you evaluate this case?

facebookresearch / EmpatheticDialogues

Dialogue model that produces empathetic responses when trained on the EmpatheticDialogues dataset.

Other

444 stars 63 forks source link

how do you evaluate this case? #26

Closed lsy641 closed 4 years ago

lsy641 commented 4 years ago

Thanks for reading this post~ I wonder how you design your human-evaluation standard? Here is a case that responsor give a responce in very empathetic way but totally not at the point. A: Oh,Shit!I fell down just now,and hurt myself. B:It must be horrible for you to stand under the sun at such a hot day. Will this dialogue get high empathy score in human evaluation?

EricMichaelSmith commented 4 years ago

Possibly - it's up to the rater's interpretation of our questions, listed in page 7 of the paper. My guess is that such a pairing would have received a low empathy score (because the response doesn't really show understanding of the feelings of the person talking about their experience) and also a low relevance score, because the response is not relevant to the original utterance.