Images, Art & Video - Wang & Kosinski 2018

[jamesallenevans]

Wang, Yilun, and Michal Kosinski. 2018. “Deep neural networks are more accurate than humans at detecting sexual orientation from facial images.” Journal of personality and social psychology 114(2): 246.

[rkcatipon]

The base assumption of the article is that it is possible to guess someone's emotional state, race, gender, and even sexual orientation by looking at their face. The hypothesis was that deep neural networks could detect facial features of gay people that were not discernible to humans.

My questions were:

1. How do the authors know that there are features that are beyond the scope of human perception? The study's classifiers may perform better than human annotaters, but that does not prove the existence of machine-only detectable gay features.
2. Did they gain the consent of the people whose face they used in their analysis? Did they anonymize the dataset? How do you anonymize faces? The authors themselves state that, "One's face... cannot be easily concealed".
3. Is the prenatal hormone theory of gayness actually widely accepted? It seems at odds with current literature on gender and sexuality that look at sexual orientation as a negotiated identity and not the result of biological predeterminants.
4. Did they actually provide any strong evidence for PHT?
5. Why is femininity used as a proxy for gayness in men? And how do the researchers define femininity and masculinity?
6. If the results of the study are generalizable beyond Caucasians, then how does the claim that women who tend to have their head covered and have dark hair are more likely to be classified as a lesbian, apply to women of color? How about women who cover their heads for religious reasons?
7. Considering the origins of their data, dating profiles or Facebook images from more openly out individuals, how do they know that the DNN model extracted features that apply to gay people and not just trends within a very particular subset of the population? For example, how can they tell if the DNN is responding to an actual latent feature as or just a popular hairstyle for a gay man on a platform?

[tzkli]

This is a very interesting paper, but the authors' study design and the implications of the study, if valid, are disturbing (as has been pointed out by @rkcatipon). Ethical issues aside, the biggest problem is that the data the authors use hardly allow them to arrive at the claims they make in the paper. The origins of both of their data sets (Facebook and dating sites) capture a subset of the LGBT population that is willing to make their profiles public online, while the human annotators have their minds "trained" with a probably more diverse sample. This finite sample bias and non-randomness in sampling haunt most applications of machine learning to empirical research and often limit the generalizability of findings. My question is, how did the authors get away with their over-generalizations?

[arun-131293]

@rkcatipon (and @tzkli ) have covered some of the core methodological issues and @rkcatipon suggests ethical issues when she says "Did they gain the consent of the people whose face they used in their analysis? Did they anonymize the dataset? How do you anonymize faces? The authors themselves state that, "One's face... cannot be easily concealed".

I think we need to ask an even fundamental question beyond anonymization, what is the motivation of such a study? What does it accomplish when you have an opaque NN that can infer sexual orientation from data? Normally, the scientific motivation which is to gain an understanding of the underlying problem(in which case we don't use Neural Networks) is enough. Their failure to justify their study other than to state they want to test Prenatal hormone theory is less than convincing. Even if that was their motivation, in reality they are just predicting orientation from facial features, not testing PHT in any meaningful way.

Secondly, they claim to "advance our understanding of the origins of sexual orientation and the limits of human perception". The fact that they might not be robustly testing human perception is already made by @tzkli above; but even more importantly the fact they think predicting orientation from facial features/gender expression will anyway lead to "advance" in "understanding of the origins of sexual orientation" reveals a radical ignorance of the biological literature on the subject of expression of complex traits. That is akin to saying "we can advance the understanding of socio-biological mechanisms underlying violence, by predicting violent behavior from facial features"; a laughable claim at best.

[cytwill]

1. The definition of “gender atypical” is ambiguous, and might be sensitive to a specific data group. For example, if we change to another dataset, can we still yield conclusions like “gay man have narrower jaws, longer noses”. Do these sort of differences widely and stably exist?

2. The authors mentioned makeup and grooming on the face, so would these facial adjustments affect the detection of fixed morphological features? (Special cases like Plastic surgery...)

3. How to decrease the risk of traits leakage in photos (like what kind of poses or lighting...)

[deblnia]

Echoing @tzkli who echoes @rkcatipon, I'm more than a little bit disturbed by both the claims in this paper. Computational methods necessarily rely on sampling (the sampling of the data that both trains and tests the model), so arriving at a general claims like "Deep Neural Nets can detect Sexual Orientation from Faces" feels misleadingly bold. Outside from this paper, though, are there ways to separate a computational method from sampling?

[skanthan95]

"Delaying or abandoning the publication of these findings could deprive individuals of the chance to take preventive measures and policymakers the ability to introduce legislation to protect people".

Are the authors claiming that their flagrant violations of participant privacy are, in a roundabout way, doing good if they act as a catalyst for introducing legislation that would not allow this kind of violation to happen again?

And, as my peers have pointed out: masculinity and femininity are highly subjective qualities that vary across cultures, making them particularly difficult to operationalize.

What are some ways (if any), that researchers can conduct research on this topic more ethically? Is it inherently impossible to, conceptually and practically (as far as protecting people's privacy goes?)