search

dccuchile / wefe

WEFE: The Word Embeddings Fairness Evaluation Framework. WEFE is a framework that standardizes the bias measurement and mitigation in Word Embeddings models. Please feel welcome to open an issue in case you have any questions or a pull request if you want to contribute to the project!
https://wefe.readthedocs.io/
MIT License
175 stars 14 forks source link

Metrics not discussed in the paper #14

Closed raffaem closed 3 years ago

raffaem commented 3 years ago

Hello,

The WEFE paper presented at IJCAI discuss WEAT, RND, and RNSB, but it does not discuss MAC and ECT.

I wondered whether there were an updated version that discussed those?

Without opening a new issue, the reference for MAC should actually be

Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings

Instead in the documentation is written as:

Black is to criminals caucasian is to police: Detecting and removing multiclass bias in word embeddings

(criminals is plural and "as" is missing: it0s a minor thing but the reference manager doesn't find it written in this way :))

pbadillatorrealba commented 3 years ago

Hello @raffaem,

In the IJCAI paper we only limit ourselves to the metrics you mention (WEAT, RND, and RNSB). ECT was not included in our paper mainly because we did not know it until then. On the other hand, we were not able to implement MAC well at the time, so we did not consider it.

In a few days I will release a new version of WEFE with MAC implemented (plus some debias methods). I also expect this version to include a script that allows to evaluate bias using all of these metrics and then update the results obtained in IJCAI.

Thanks for your indication of the error in the paper Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. I will keep it in mind to correct it!

In case you find it useful, you can find the MAC implementation in the multiclass_evaluation function of https://github.com/TManzini/DebiasMulticlassWordEmbedding/blob/master/Debiasing/evalBias.py. On the other hand, ECT is proposed in the paper: Attenuating Bias in Word Vectors You can find the original implementation here: https://github.com/sunipa/Attenuating-Bias-in-Word-Vec as well as at https://github.com/dccuchile/wefe/blob/master/wefe/metrics/ECT.py

Best regards, Pablo.