Add Gender option for Other

thanks!

On Thu, Mar 17, 2016 at 4:25 PM, Shaun Wallace notifications@github.com wrote:

Closed #77 https://github.com/brownhci/drafty/issues/77.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#event-594011553

How will we (or crowdworkers) know if they are not Male or Female, and I wonder if this is easy to get wrong? I wonder whether this something that someone besides people who know that professor personally would know. Basically, is there a way to get the ground truth for this field?

Valid concerns.

We are making folks guess someone's gender from a photo or name, which on the surface seems problematic. BUT... if what we are evaluating for is biases based on perceived gender along the traditional gender binary, that means that Drafty is working best if we see what the "crowd" agrees on as someone's gender.

I don't love that there is not a third option, but I see why an "Other" might not make actually make sense in this case. I guess this is just a problem that goes along with collecting gender as a variable in the first place.

I guess we should undo this change then, but thanks for considering it! I'll think on this.

On Thu, Mar 17, 2016 at 5:28 PM, Jeff Huang notifications@github.com wrote:

How will we (or crowdworkers) know if they are not Male or Female, and I wonder if this is easy to get wrong? I wonder whether this something that someone besides people who know that professor personally would know. Basically, is there a way to get the ground truth for this field?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198088652

Jeff,

Switched it back.

Quick question what was the original motivation for asking for Gender for thee original class assignments? Why was it included in 2015 but not 2014? For the data analytics having gender information would provide some excellent analysis in hiring trends that people would be interested in.

There is also a flipside to the argument to include "Other" as an option. Remember we are asking someone else to identify someone's gender. Which is a tricky question to ask as Jeff pointed out. What happens if someone is identified on Drafty as "Other" and that offends them bc it is not how they self identify.

There might not be a perfect way to handle asking gender and/or race in Crowd Sourcing. Trying to find other examples or scholarly articles relating to this but I have not found anything yet.

Question if someone on the list wants their Gender not to be identified should we disable the ability to suggest Gender for them?

On Thu, Mar 17, 2016 at 5:28 PM, Jeff Huang notifications@github.com wrote:

How will we (or crowdworkers) know if they are not Male or Female, and I wonder if this is easy to get wrong? I wonder whether this something that someone besides people who know that professor personally would know. Basically, is there a way to get the ground truth for this field?

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198088652

Shaun Wallace Senior Application Developer, ICERM http://icerm.brown.edu/ Brown University 121 S. Main St. Providence, RI 02903 401-863-7635 shaun_wallace@icerm.brown.edu Shaun_Wallace@icerm.brown.edu

Yea this is tricky. I totally get your point, misusing the "Other" option could be just as insulting for people. And "Other" isn't really a gender identity anyway, just a catch-all. The option of disabling is worth thinking about. Chances are no one will actually be bothered by us collecting this data, but if they are we should obviously be respectful of that. It's a bit weird to have the crowd fill in such a personal, rather than professional, variable.

I'd be curious to see if this is tackled in other crowd-sourcing platforms, but I wouldn't be surprised if it isn't usually seen as an issue - a lot of psychology and social science uses gender as a binary variable all the time in analysis, and because there are a lot of differences in outcomes that are correlated with gender this can be very useful. For example, I'm sure people will want to learn about female hiring trends in computer science departments.

On Thu, Mar 17, 2016 at 6:30 PM, Shaun Wallace notifications@github.com wrote:

Jeff,

Switched it back.

Quick question what was the original motivation for asking for Gender for thee original class assignments? Why was it included in 2015 but not 2014? For the data analytics having gender information would provide some excellent analysis in hiring trends that people would be interested in.

There is also a flipside to the argument to include "Other" as an option. Remember we are asking someone else to identify someone's gender. Which is a tricky question to ask as Jeff pointed out. What happens if someone is identified on Drafty as "Other" and that offends them bc it is not how they self identify.

There might not be a perfect way to handle asking gender and/or race in Crowd Sourcing. Trying to find other examples or scholarly articles relating to this but I have not found anything yet.

Question if someone on the list wants their Gender not to be identified should we disable the ability to suggest Gender for them?

On Thu, Mar 17, 2016 at 5:28 PM, Jeff Huang notifications@github.com wrote:

How will we (or crowdworkers) know if they are not Male or Female, and I wonder if this is easy to get wrong? I wonder whether this something that someone besides people who know that professor personally would know. Basically, is there a way to get the ground truth for this field?

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198088652

Shaun Wallace Senior Application Developer, ICERM http://icerm.brown.edu/ Brown University 121 S. Main St. Providence, RI 02903 401-863-7635 shaun_wallace@icerm.brown.edu Shaun_Wallace@icerm.brown.edu

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198109554

I wasn’t truly saying we shouldn’t have an “other”, but maybe that this would be hard for a crowd worker to decide. The email the spurred this field is from CMU:

"I'm exploring the possibility of looking at gender issues in publishing. I'd like to potentially expand your data with new columns (at a minimum, gender and graduation year, but I'm exploring other possibilities as well). In addition, I'd like to link faculty to DBLP or some other bibliography (e.g. see http://www.cs.rice.edu/~sc40/Rankings/ http://www.cs.rice.edu/~sc40/Rankings/)

Let me know if you think it would be possible for me to help extend the data.“

from Professor Mankoff at CMU https://www.hcii.cmu.edu/people/jennifer-mankoff https://www.hcii.cmu.edu/people/jennifer-mankoff

Yeah it’d be interesting to see what other crowd platforms do for this field.

Jeff

On Mar 17, 2016, at 7:58 PM, Lucy Van Kleunen notifications@github.com wrote:

Yea this is tricky. I totally get your point, misusing the "Other" option could be just as insulting for people. And "Other" isn't really a gender identity anyway, just a catch-all. The option of disabling is worth thinking about. Chances are no one will actually be bothered by us collecting this data, but if they are we should obviously be respectful of that. It's a bit weird to have the crowd fill in such a personal, rather than professional, variable.

I'd be curious to see if this is tackled in other crowd-sourcing platforms, but I wouldn't be surprised if it isn't usually seen as an issue - a lot of psychology and social science uses gender as a binary variable all the time in analysis, and because there are a lot of differences in outcomes that are correlated with gender this can be very useful. For example, I'm sure people will want to learn about female hiring trends in computer science departments.

On Thu, Mar 17, 2016 at 6:30 PM, Shaun Wallace notifications@github.com wrote:

Jeff,

Switched it back.

Quick question what was the original motivation for asking for Gender for thee original class assignments? Why was it included in 2015 but not 2014? For the data analytics having gender information would provide some excellent analysis in hiring trends that people would be interested in.

There is also a flipside to the argument to include "Other" as an option. Remember we are asking someone else to identify someone's gender. Which is a tricky question to ask as Jeff pointed out. What happens if someone is identified on Drafty as "Other" and that offends them bc it is not how they self identify.

There might not be a perfect way to handle asking gender and/or race in Crowd Sourcing. Trying to find other examples or scholarly articles relating to this but I have not found anything yet.

Question if someone on the list wants their Gender not to be identified should we disable the ability to suggest Gender for them?

On Thu, Mar 17, 2016 at 5:28 PM, Jeff Huang notifications@github.com wrote:

How will we (or crowdworkers) know if they are not Male or Female, and I wonder if this is easy to get wrong? I wonder whether this something that someone besides people who know that professor personally would know. Basically, is there a way to get the ground truth for this field?

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198088652

Shaun Wallace Senior Application Developer, ICERM http://icerm.brown.edu/ Brown University 121 S. Main St. Providence, RI 02903 401-863-7635 shaun_wallace@icerm.brown.edu Shaun_Wallace@icerm.brown.edu

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198109554

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/brownhci/drafty/issues/77#issuecomment-198137212

Thanks for digging that up Jeff. She has some excellent research. With this issue I am of the opinion that either way is fine. Trying to look at from different angles. It is a complex decision. I believe having the gender is very important for the analysis.

So searching through Google Scholar I found 1 paper that actually ran an experiment to see how accurate workers on Mechanical Turk were at identifying someone's gender from a picture:

section:* 4.2 Gender Classification* http://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=6920084#sec3 The paper was published in June 2015

From the paper accuracy rates: Correctly identifying Males: 99.6% Correctly identifying Males: 93.5% - 94.1%

Workers only had two options to pick from: Male or Female

So we at least have a published source to confirm that workers can accurately identify someone's gender. Although I did not see any studies where the gender classifications used have been expanded.

We could make it so if someone wants to identify someone's Gender they need to provide a photo URL?

Non CS paper on Gender Classifications and legal cases: http://gas.sagepub.com/content/24/6/814.abstract

From the paper:

"Courts conceptualized legal gender in three distinct ways: descriptively, as a feature of individual identity (and as a metric for identity verification); relationally, as a characteristic of specific gendered relationships; and finally hierarchically, as a construct to which power attaches."

Drafty certainly falls into the "a feature of individual identity (and as a metric for identity verification)" category.

So after all that I think as of now we are ethically and scientifically OK.

Hope everyone has a good Friday, ~Shaun

brownhci / drafty

Add Gender option for Other #77