ECT score

[efatmae]

Hi,

I'm using the WEFE package to measure bias in different word embeddings. However, When I use ECT the results are not clear to me. AS mentioned in the original paper "Attenuating Bias in Word vectors." they explain that the higher the ECT score the better its s meaning that the higher the score the less biased a word embedding is. I'm wondering if you are taking that in consideration in the Ranking process?

Thanks, Fatma

[pbadillatorrealba]

Hi @efatmae,

I'm going to assume that you are asking about the embeddings bias ranking process that we described in the WEFE paper.

The answer is that at the time we were developing the WEFE case study we did not contemplate ECT mainly because we were not aware of its existence. ECT was implemented after we published the package by @mspl13. Therefore, the results reported in the paper do not include this metric.

Unfortunately the create_ranking utility was originally intended for metrics where 0 is the ideal (or unbiased) value, and while create_ranking has an ascending parameter, its functionality is limited to indicate how to calculate all rankings and does not allow to specify what to do with each score set per metric.

During this month I plan to release a new version. In this new release I could include a solution for this issue, maybe in a parameter that defines how to create the rankings from the given values.

Hopefully I have been able to answer your question! (if not, I have no problem to answer again :) ).

Best regards, Pablo.

[efatmae]

Thanks for your answer.

On Thu, Jul 8, 2021 at 8:16 PM Pablo Badilla @.***> wrote:

Hi @efatmae https://github.com/efatmae,

I'm going to assume that you are asking about the embeddings bias ranking process that we described in the WEFE paper.

The answer is that at the time we were developing the WEFE case study we did not contemplate ECT mainly because we were not aware of its existence. ECT was implemented after we published the package by @mspl13 https://github.com/mspl13. Therefore, the results reported in the paper do not include this metric.

Unfortunately the create_ranking utility was originally intended for metrics where 0 is the ideal (or unbiased) value, and while create_ranking has an ascending parameter, its functionality is limited to indicate how to calculate all rankings and does not allow to specify what to do with each score set per metric.

During this month I plan to release a new version. In this new release I could include a solution for this issue, maybe in a parameter that defines how to create the rankings from the given values.

Hopefully I have been able to answer your question! (if not, I have no problem to answer again :) ).

Best regards, Pablo.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dccuchile/wefe/issues/13#issuecomment-876682235, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLVAXIZTHRER2EEDLONDXDTWX2RVANCNFSM5AA3H3FQ .

-- Best Regards, Fatma Amin Elsafoury

[pbadillatorrealba]

Hello @efatmae ,

After a couple of weeks of thinking about a good solution, I couldn't finally come up with a good solution.

The best course of action I can think of is:

1. Calculate the rankings separately. This would involve calculating rankings of the WEAT, RND, RNSB, etc... results with:

ranking_general = create_ranking(results_general, ascending=True)

and those of ECT with:

ranking_ect = create_ranking(results_ect, ascending=False)

Then, make a concatenation of the generated dataframes:

rankings = pd.concat([ranking_general , ranking_ect], axis=1)

2. Indicate with a warning in the userguide that ECT must be calculated separately.

Sorry I don't have a good solution for this problem.

Best regards, Pablo.