Exploring Semantic Spaces - Garg, Schiebinger, Jurafsky & Zou 2017

[jamesallenevans]

Nikhil Garg, Londa Schiebinger, Dan Jurafsky and James Zou’s follow-on article, 2017. “Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes.” PNAS 115 (16) E3635-E3644.

[heathercchen]

This is a very insightful analysis based on a huge volume of data. My questions are about the methods the authors used to analyze stereotypes in women's occupations in section 3.1 (at p.5). The authors proposed that

... the embeddings could reflect additional social stereotypes beyond what can be explained by occupation participation.

1. The authors collected crowdsource scores to reflect aggregate human judgment as to whether an occupation is stereotypically associated with men or women. Then they further looked at the embedding bias extracted from the corpus to see whether it is correlated with the crowdsource scores. Why they are reporting r^2 for correlations here? Shouldn't it be a correlation index rho instead?
2. Why the authors used joint regression with crowdsource scores and occupation log proportion as the dependent variable and embedding bias as the independent variable? If the only need is to test whether these three variables are correlated with each other (as the signs of the coefficients are not important, and obvious), a simple F-test might be a better choice.

[Dominiquo]

I'm not sure what the authors mean when they say they want to "debias" words. I get that there is need to make sure "problematic" associations don't perpetuate, but what would this mean on the scale of the entire word space when each word's position is defined via context and relative to other words?

From Article: "An important direction of research is on developing algorithms to debias the word embeddings."

This seems more like a call to action right now, which I guess can remain more speculative than prescriptive, but I'd like to discuss more in depth what this would even mean in practice outside of applying some type of more rigid rule system on top of the existing semantic structure.

[di-Tong]

To quantify the change in bias, the authors train an embedding model for each decade. I wonder what is the criteria for choosing an appropriate unit of analysis for comparison? In real research process, should we tune between different levels of unit to see which one produce more stable results? In addition, what is the strength of this design in comparison with dynamic word embedding?

[rachel-ker]

It was rather innovative to be making use of the bias in the embeddings to quantify stereotypes and I found it interesting that the word embeddings were tracking significant events. As the author mentioned, it is an exploratory analysis. I wonder if this study could be extended to identify the types/characteristics of movements/events/laws that were more associated with changing stereotypes? and others that did not impact bias/stereotypes? Perhaps that could give us some clues to identify how these biases evolve.

[luxin-tian]

In this research, the authors make lists of "neutral words" as the words of interest to measure bias in terms of gender and ethnicity. I wonder how did the authors decide whether a word is "neutral" in the sense of gender and ethnicity? It is reasonable that words such as "fireman", "mailman" were excluded, but did they coded the lists of words by hand or is it possible that any classification method can be used?

[wunicoleshuhui]

The authors distinguished race and ethnicity by using the different distributions of last names based on the ethnicity, and thus they had to exclude African Americans from the analysis due to similar last names between white and African Americans. How can this pitfall be addressed in another embedding model study? Is it possible to do another study based on first names between the two groups?

[laurenjli]

In this paper, authors use word embeddings to quantify social change but can we also use word embeddings to explain why? Like why physical appearance hasn’t changed as much as competence? Or do we need to rely on external historical knowledge to do this as is done in the paper?

[skanthan95]

The authors state:

"First, we use women and minority ethnic participation statistics (relative to men and Whites, respectively) in different occupations as a benchmark because it is an objective metric of social changes".

I agree that this is a very strong predictor/gauge of social change, but I'm wondering if it can appropriately track social changes that specifically pertain to shifts in gender-role perception (since this is what the authors are interested in- (IV) shifts in gender and race perception, operationalized through text analysis methodology (word embeddings) -> (DV) how participation in the workforce changes over time by occupation and race). In other words, I can imagine other roadblocks (in tandem with, instead of, or in addition to) gender stereotypes impacting whether or not women participate in certain jobs. Have I misunderstood the authors' rationale ?

[arun-131293]

Looking at both this reading and the orienting reading of this week, is it fair to say that word embedding based studies are mainly used to fish out word associations (which words/concepts) are associated with which other words given a large enough unit of text? If that's the case, it throws doubt on if such methods can be used for any micro-level analysis beyond the broad patterns discovered here(like the changing association of "Islam" and "Terrorism".)

[yirouf]

This is a particularly interesting read. It is intriguing to look at word embedding from a cultural and a linguistic view at the same time. I do think this will be a very useful tool to track historical changes that was displayed in usage of certain word's meaning (or change of a word's meaning) AKA bias. I'm also curious about how the criteria used to train is determined?

[bjcliang-uchi]

as @luxin-tian asks I am also concerned about neutrality. Also, isn't it that using last name to detect ethnic groups can create a bias towards men and single women?

[rkcatipon]

This was an interesting read, but like @laurenjli I was also inclined to wonder why shifts took place. For example, with Hispanic bias embeddings, we see a decline and then a sharp increase .

I wondered if we could use the same word embedding technique to look for major events in the Google news corpora that might have influenced a shift in perception to a racial group? For example, the Zoot Suit Riots in 1943 were indicative of racial tensions in LA and were commented on extensively by the media. Perhaps we could look at news coverage of the event and find a way to track its effect on HIspanic bias embeddings, or at the very least, critically analyze if a correlation exists at all.

[YanjieZhou]

I am curious about the reasonable range of so-called neutral words in terms of ethnical and sexual biases. Some words that include the use of "man" actually results from language conventions, which are used without any consicousness about biases. I am wondering whether to take the use of neutral words in these scenarios will lead to errors in the results.

[meowtiann]

I think this goes back to egg and chicken problem. If not having great theories before getting into thee data, the outcome is less interesting. But using this method to generate theory really requires a lot of knowledge into this field, which might rely on other theories. This paper has great,interesting data and results but not in a sense for generating a theory for gender and ethnicity studies. Unless a gender or ethnicity scholar uses the same method.

[ziwnchen]

When the author tries to explain the gender stereotype shifts in the word embedding space using historical events, is it valid to say that certain changes are caused by those events? Since the corpus mainly consists of paper-based text in the past, is it possible that there are other confounding factors and publishing time lags that may allow different explanations?

[kdaej]

Although this paper does not seek the cause of such a trend it found, I was curious if there is a way to identify social factors or policies at that time influencing the written texts used in the study. The study mentions that Asians were associated with being “barbaric”, “hateful”, “monstrous”, and “bizarre” before 1950. This may be explained by the fact that the U.S. was at war with Japan in the 1940s. However, it is not clear if there is a link between these texts and the government policy toward Japanese Americans during this time. I wonder if the government’s attitude toward immigrants affected people to write more aggressively about Asians in general.

[VivianQian19]

Garg et al. use word embedding models to quantify the shifts in gender and ethnic stereotypes. The authors note that their goal is quantitative descriptive analysis rather than causal explanations. I’m wondering how and if it’s possible for word embedding models to explore causal relations?