search

dccuchile / wefe

WEFE: The Word Embeddings Fairness Evaluation Framework. WEFE is a framework that standardizes the bias measurement and mitigation in Word Embeddings models. Please feel welcome to open an issue in case you have any questions or a pull request if you want to contribute to the project!
https://wefe.readthedocs.io/
MIT License
173 stars 14 forks source link

Missing WEAT words #52

Closed kato8966 closed 2 months ago

kato8966 commented 11 months ago

There are two versions of unpleasant_5 words in the original WEAT paper. One of them is used in WEAT 1, and the other is used in WEAT 3. (For example, the former includes "prison," while the latter does not.) However, wefe/datasets/data/WEAT.json has only the former.

I fixed the problem by renaming unpleasant_5 to unpleasant_5a and adding unpleasant_5b (This is the fix branch. https://github.com/kato8966/wefe/tree/fix-weat-words), but this breaks backward compatibility.

Should I proceed and make a PR, or should I try a different approach?

pbadillatorrealba commented 10 months ago

Hi @kato8966 ,

Thanks for noticing it and trying to correct it! Yes, make a pull request to develop and we will evaluate it. Most likely we will have to change the experiments notebook.

kato8966 commented 10 months ago

Sorry, but I'm currently busy and don't have enough time to fix all the problems caused by this compatibility-breaking change. What do you think I should do?

pbadillatorrealba commented 10 months ago

No problem :) Just upload the patch with the words and I will correct the compatibility issues later.

felipebravom commented 2 months ago

Solved