search

kayburns / women-snowboard

Apache License 2.0
18 stars 4 forks source link

gender classification out of caption #24

Open DavidHuji opened 2 years ago

DavidHuji commented 2 years ago

Thanks for sharing the code, may I ask how did you measure the gender error rate? I.e. how did you classify the gender when the original model only produce captions? did you train another classifier on top of the captions? It is a bit unclear from the paper. Tahnks!

LisaAnne commented 2 years ago

If I understand your question correctly, we did this by word matching. E.g., if gt captions all include male generated words, we label this as a "male" caption then look at the words in our generated caption to see if any of them include male gendered terms. Hope that helps!

On Thu, Aug 4, 2022 at 5:18 PM David Nukrai @.***> wrote:

Thanks for sharing the code, may I ask how did you measure the gender error rate? I.e. how did you classify the gender when the original model only produce captions? did you train another classifier on top of the captions? It is a bit unclear from the paper. Tahnks!

— Reply to this email directly, view it on GitHub https://github.com/kayburns/women-snowboard/issues/24, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKCNWS66PG26EXHJRYXSEDVXPUM5ANCNFSM55TENPUQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

DavidHuji commented 2 years ago

Oh, I see, thanks!

DavidHuji commented 2 years ago

Could you please explain also one thing regarding the metrics you have used?

You have defined the error rate metric as "the number of man/woman misclassifications, while gender neutral terms are not considered errors", and in addition, you have defined another metric for gender ratio as "the ratio of sentences which belong to a 'woman' set to sentences which belong to a 'man' set".

Now, I think those metrics won't catch an algorithm that will always return gender-neutral terms, right? such an algorithm will get ideal results of zero error rate, and a gender ratio of 1. Am I missing something?