search

nyu-mll / crows-pairs

This repository contains the data and code introduced in the paper "CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models" (EMNLP 2020).
102 stars 24 forks source link

Score for antistereotype pairs question #8

Closed Lynx1820 closed 2 years ago

Lynx1820 commented 2 years ago

In line 34-39, sent_more and sent_less get flipped if they have antistereo direction, but when calculating the score for the antistereo direction, you compare if score['sent2_score'] > score['sent1_score']: in line 239, essentially flipping them back and doing the same comparison as with the stereo direction. Is this right? To my understanding, you want to flip the comparison for pairs with antistereo direction, since the stereotypical sentence is sent_less in these pairs. 

if direction == 'stereo': 
  sent1 = row['sent_more']
  sent2 = row['sent_less']
else:
  sent1 = row['sent_less']
  sent2 = row['sent_more']
Lynx1820 commented 2 years ago

never mind, I misunderstood and it's "sent_less" the sentence that violates the stereotype.