Week 4 - Possibility Readings

[bhargavvader]

Post a reading of your own that uses deep learning for social science analysis and understanding, with a focus on text data.

[Raychanan]

Title: Deep Learning-Based Document Modeling for Personality Detection from Text

This article presents a deep learning based method for determining the author's personality type from text: given a text, the presence or absence of the Big Five traits is detected in the author's psychological profile. For each of the five traits, the authors train a separate binary classifier, with identical architecture, based on a novel document modeling technique. Namely, the classifier is implemented as a specially designed deep convolutional neural network, with injection of the document-level Mairesse features, extracted directly from the text, into an inner layer. The first layers of the network treat each sentence of the text separately; then the sentences are aggregated into the document vector. Filtering out emotionally neutral input sentences improved the performance. This method outperformed the state of the art for all five traits, and the implementation is freely available for research purposes.

Suggesting how its method could be used to extend social science analysis: I don't know if this could be considered a possible extension of social science, but my intuition is to use this method to determine the type of likelihood of developing depression (high, medium, low). Since depression is highly correlated with a person's personality, I think this method might be useful in this case as well.

Describing what social data you would use to pilot such a use: I think the main concern is personal privacy. So I would say that we could download all their online content on Twitter, Facebook, Instagram, etc., with people's consent, and build a model in combination with the corresponding user's history of depression. After that, the model would be used to make predictions about the probability level of a person suffering from depression.

[william-wei-zhu]

Title Recommendation of startups as technology cooperation candidates from the perspectives of similarity and potential: A deep learning approach

Summary: How can deep learning algorithms help tech companies identify ideal startups to acquire? This paper applies two methods to generate a recommendation system for tech companies in the VR space: (1) the researchers utilized doc2vec to extract feature information from crunchbase texts for startups and patent abstracts for the acquiring companies. Then they generated technological similarity scores between startups - acquiring company pairs. Where high similarity scores indicate potential for exploitative acquisition, and low similarity scores indicate potential for exploratory acquisitions. (2) the researchers used factor analysis to evaluate startups’ potential based on crunchbase data. High score indicates that the companies are desirable for acquisitions.

Suggesting how its method could be used to extend social science analysis I am interested in using the first method, doc2vec, to measure the cultural similarity between startups - acquiring company pairs based on glassdoor company reviews. What types of cultural similarity/differences can predict positive synergy?

describing what social data you would use to pilot such a use I plan to use glassdoor’s employee review data.

[k-partha]

Title: SemEval-2019 Task 3: EmoContext, Contextual Emotion Detection in Text

Summary: This paper (by Microsoft Research) describes an emotion detection task that involves classifying annotated conversations with an agent as either 'Happy', 'Sad, or 'Angry' using the context of the conversation. It reviews the performance of various models submitted by competing teams to produce the best overall results. Emotion detection, especially only using text (without voice or facial cues), is a highly challenging task that is likely highly advantaged by knowledge of the context of the conversation. This makes the application of contextual-embedding text models to this task an exciting prospect. The paper finds that ensemble models that leverage BERT and Elmo are some of the best performing models and that Bi-LSTMs are the most common.

Suggested expansions to social science analysis: Analysis of discourse on social media is one example where emotion detection can reveal an interplay of emotional states in the course of agreement/disagreement and heterogeneities in the engagement styles of different demographics/socio-political factions (just how angry are millennials? Is there a lifecycle of emotional states in the course of political engagement?). Countless other social science analyses could leverage deep-learning-based contextual emotional detection.

Possible social data pilot: Twitter tweet engagement data: Assessing the support/opposition to controversial political issues by analyzing the range of emotional responses attracted by tweets by central actors in that political theme (on Twitter). How many sad, angry, and happy replies are tweets supportive of a particular political agenda getting? This information can help us understand the general population-level support for various political issues (after post-stratification) and understand the public's overall response to certain social ideas (How does the public feel about GMO advocacy now? How has engagement changed over the past X time duration?). We would need to use the Twitter API to obtain the data and train and deploy contextual-embedding based classifiers for analysis.

[pcuppernull]

Seckin, Tugkan and Zeynep Hilal Kilimci. 2020. "The Evaluation of 5G technology from Sentiment Analysis Perspective in Twitter."

Summary: This paper auditions several types of models for a text-based classification task and find that deep learning models far outperform traditional machine learning models. Using a corpus of approximately 60,000 tweets that use the hashtag “#5G”, the authors seek to build a classifier to label tweets as either positive or negative and ultimately measure the perception of 5G technology among Twitter users. After labeling the tweets as either positive, negative, or neutral and keeping only the positive and negative tweets, the authors construct a variety of models to classify the tweets, including: naive bayes, support vector machines, k-nearest neighbors, recurrent neural networks, and long short-term memory networks. The experimental results indicate that the deep learning approaches vastly outperform the standard machine learning approaches – for example, the best machine learning model (linear SVM) achieved test classification accuracy of 75.1% while the best deep learning model (LSTM) reached a test classification accuracy of 81.5%. The authors indicate that when social media is used as a tool to measure attitudes towards contentious issues, researchers and policymakers should use deep learning models to obtain the most accurate understanding of public opinion.

Extension to social science: While this paper is simple in its approach, similar studies could be performed to encourage social scientists to embrace deep learning models. Many machine learning methodologies that are outdated by newer technologies are still commonplace in political science, for example, and deep learning researchers could replicate popular studies that use older methods to demonstrate how research could be improved by using the tools of deep learning.

Application with new data: A study could extend the simple classification task above to a multiclass setting. Mitts (2019) engages in a supervised learning task that classifies tweets as demonstrating support ISIS in various ways, including: expressing sympathy for ISIS, discussing the war in Syria, or exhibiting anti-Western hostility. This paper relies on an elastic net model for classification, which could possibility be improved by employing a deep learning model.

Mitts, T. (2019). From Isolation to Radicalization: Anti-Muslim Hostility and Support for ISIS in the West. American Political Science Review, 113(1), 173-194.

[Yilun0221]

Title: Text Understanding from Scratch

Summary: This paper by Drs. Xiang Zhang and Yann Lecun aims to achieve text understanding without any prior knowledge about the text data, whether syntactic or semantic. Two ConvNets are used, a large one and a small one, each with 9 layers, “6 convolutional layers and 3 fully-connected layers”. With a given dataset, the researchers did synonym replacement using WordNet and then put them into the ConvNets. The researchers used several datasets to achieve different understanding goals. DBpedia is used to conduct ontology classification, Amazon Reviews are used for sentiment analysis and Yahoo! Answers are used for topic classification. In addition, the researchers also used news written in English and Chinese to do news categorization (whether the news is about business, sports, etc.). The experiment results show that ConvNets can do well in text understanding without prior knowledge in syntactic or semantic aspect of a certain language.

Expansions to social science analysis: ConvNets can do further than the tasks tried in this paper. Besides NER and POS tagging as the paper mentions, ConvNets can also be used to establish social networks beyond different entities and conduct public opinion analysis. In addition, this methodology can be used to predict judicial decision towards different cases based on previous decisions. In addition, it can also be used in the application of AI in medicine. For example, based on the previous symptoms or descriptions of patients and the prescriptions given by the doctors, AI can learn how to give medical suggestions in response to future medical consultation. However, this methodology is limited to cases where previous data provides solutions. If it is an uncommon case or a case that has never appeared previously, I don't think ConvNet will work...

New dataset exploration: I want to try how this methodology performs in analyzing judicial documents and medicine prescriptions. For judicial documents, there are some websites which legally gives the public access to some of them, and we can make use of these websites. For medicine descriptions, we can use the dialogues in the format of text, video or audio on medicine App.

[cytwill]

Title: MDR Cluster-Debias: A Nonlinear Word Embedding Debiasing Pipeline

Summary: This paper introduces a new pipeline, MDR Cluster-Debias, to help reduce the bias trained in general word embeddings. In particular, the authors pointed out that existing debiasing methods fail to reduce the bias at a cluster level (instead of a single direction) since there are still observable differences around the opposing seed terms (man vs woman) in the embedding space after deploying these methods. Their new method involves two steps: 1) a post-processing procedure, which is used to re-embed original word vectors into a new space via a manifold learning algorithm (MDR) 2) a cluster-based debiasing method that first uses the directional bias between the seed words to detect the top k biased words, then uses these most biased words to fit PCA and extract the first PC for all other word embeddings, and finally debiases all the embeddings by removing the PC through matrix computation.

The evaluation tasks are processed with up-stream and down-stream tasks, where the MDR Cluster method is found to have significantly superior performance than others in all up-stream tests and can maintain semantic information as good as the original embeddings as well. The new method only shows limited advantages in the downstream task, which the authors regard as new potentials to refine their definition of bias and the whole algorithm.

Extension to Social Science: The bias generated from word embeddings has been an important topic for both NLP and social science communities. On the one hand, the bias revealed by the embeddings can help us to detect and understand the disparity and stereotypes in people’s ideology towards concepts like male vs female, democracy vs republic, white vs black, etc. On the other hand, the bias existing in these models might bring unwanted consequences in social media and recommendation systems that make certain people feel being offended. We might use this framework to perform some A/B testing on online recommendation systems to see how biased and unbiased embeddings can shape different user experiences. Also, we might especially use the cluster-based bias measurement to evaluate the degree of bias in different online communities as well as the temporal change of such bias.

New dataset exploration: This MDR Cluster-Debias can be used for many different text data, including newspaper reports, books, advertisements and tweets, and comments. Using the debiased embeddings, we might even come up with some recommendations on how to rephrase a document to make it less discriminative.

[nwrim]

Cross-lingual Language Model Pretraining 2019. Alexis Conneau, Guillaume Lample. From NeurIPS 2019

1) briefly summary of the article In this paper, the authors introduce a new objective for cross-language pretraining of Transformer models: Translation Language Modeling (TLM). This is an extension of Masked Language Modeling (MLM) used in BERT. The main idea behind TLM is that it uses parallel (translated) sentences from two languages as input rather than sentences from one language. Importantly, when predicting the masked word of language A, the model can attend to both the sentence from language A and the parallel sentence from language B. This encourages the model to learn the representations of both languages and align them at the same time. The model is trained both in MLM and TLM in an alternating fashion. The authors show that this new model outperforms SOTA models in cross-lingual classification tasks and translation tasks and show that cross-lingual word embedding shows much better quality than previous models. However, I do want to note that the model requires a dataset of parallel sentences, which will be very difficult to obtain for some languages.

2) suggestion on how its method could be used to extend social science analysis I think a cross-lingual model could be greatly beneficial to social science analysis in general since we can (at least partially) mitigate the dominance of studies that only focuses on English and English-speaking cultures (e.g., by utilizing the translational capacity of the trained model). The better quality of cross-language word embedding (compared to cross-language word embeddings from an alignment of multiple models trained in one language each) could also facilitate interesting cross-cultural analysis. Multiple studies tried using cross-language word embedding to see if there are cultural similarities or differences between cultures that do not utilize the same language. It will be interesting to see if the embedding from this model also shows similar results from those studies. Also, it will be interesting to see if the bias observed in language embeddings from a model trained in a single language can be corrected if the model is trained parallelly with another language that does not show great bias on that topic.

3) describing what social data you would use to pilot such a use For the last example (bias correction), maybe we can use two separate news corpora, each with a different language, and the translation of that corpus (so there will be four corpora) where the bias pattern is different (e.g., the countries where each language is dominant have very different racial properties). Then, we can see if the bias found, like the work by Dr. Caliskan, still exists strongly on the new embedding space.

[ajahn6]

Modeling and Measuring Expressed (Dis)belief in (Mis)information CWSM 2020, Jiang, Metzger, Flanigan & Wilson

Summary: Jiang et al. investigate misinformation on social media by characterizing prevalence, time effects, influence of fact-checking, and cross-platform variation. Misinformation claims were hand-coded based on Politifact data and traced these claims to their original tweets. Retweets and comments were extracted and labeled based on their expression of belief or disbelief (or neither). They analyzed their corpus of tweets with several different language processing approaches, including LIWC and ComLex lexicons as well as neural net models. Of all approaches, neural learning models performed best, with ROBERTa slightly beating out BERT and XLNet. The tuned ROBERTa model was then used to identify prevalence of belief and disbelief at scale. Jiang et al.'s model found, across 2.6 million tweets associated with identified misinformation, that expressed belief was always more represented than expressed disbelief, but that belief and disbelief varied relative to actual veracity of the message. They also found that addition of fact-checking had a slight but significant effect in decreasing belief of false claims. Finally, Jiang et al. found variance across platforms, but did not delve deep into the subject.

Use in Social Science analysis: Jiang et al.'s paper tackles a social science problem concerning the character of the spread and acceptance of misinformation. Their tuned ROBERTa model could be useful for other researchers seeking to perform similar analysis on other misinformation subjects or across platforms, and testifies to the effectiveness of deploying neural net language processing models for social media data analysis at scale. It may be beneficial to incorporate Jiang et al.'s tuned model into a comparison of other trained models to assess performance before deploying the model for classification of substantially different discursive environments where language norms might be different, but Jiang et al.'s model training may be robust enough that these potential problems wash out at scale.

Exploration with new data: Deploying Jiang et al.'s model on radicalization communities and conspiracy networks could be useful for characterizing shifting attitudes towards certain topics over time. For instance, scraping tweets with language associated with attitudes towards vaccines could reveal shifting patterns of (dis)belief over time. This information could be pared with demographic data for a more robust picture of who believes what when. Such information would be a useful jumping off point for considering the influence of (social/broadcast) media, political decisions, or other events on belief in misinformation.

[bakerwho]

Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change William L. Hamilton, Jure Leskovec, Dan Jurafsky

1. Summary The authors use text corpora in 4 languages and spanning 2 centuries to evidence quantitative laws of semantic change. The first is the law of conformity, finding that the rate of semantic change scales with an inverse power-law of word frequency. The second is the law of innovation, stating that independent of frequency, words that are more polysemous (having multiple meanings) have higher rates of semantic change. They use word embedding methods based on SVD (singular value decomposition), PPMI (positive point-wise mutual information) and Word2Vec.

2. Questions to motivate future work How do word embedding adapt over time in other canonical cases (such as with news data)? How do individual words respond to each other as they change? Are there emergent changes? Can we operationalize other social phenomena within the mechanics of these changes?

More technically: Do the schema of sampling and training make a difference to these evolutions, and if so, how much?

3. Social data to use to pilot this The NOW (News On the Web) corpus has 10 years of news data from over 10 countries, indexed by source. We could use this to trace differences in word usage across countries, or across news channels.

[jsoll1]

Title: Identifying unreliable online hospitality reviews with biased user-given ratings: A deep learning forecasting approach

Summary: This paper's core purpose is to tell which reviews on tourism sites are suspect and biased. Their process is to first find the relevant features through textual analysis to gather emotional words. Afterwards they create a model on the feature extraction to find inconsistencies between the textual review itself and the score given. They tried three different kernals: FC, CNN, and LSTM models.

Use in social science analysis: I think that there are many sectors in social science where it's important to find inconsistencies between actions and sentiments. A lot of judgement and decision making research is focused on smaller studies exploring inconsistencies in behavior and thoughts. Tools which can extend that to big data seem to be a potential expansion to the field.

Exploration with new data: I'm interested in seeing this kind of tool used on IMDB reviews. Intuitively it seems as if people would rate movies numerically based on their expectation of what their community wants it to be rated as, while they would give a more honest and personal take in their review. I think that if these kinds of inconsistencies are found then that would validate that hypothesis, which is extremely interesting

[luxin-tian]

Demography with deep learning and street view Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, Li Fei-Fei

Summary: This research shows that deep learning approaches can be used to infer socioeconomic attributes such as "income, race, education, and voting patterns" based on cars detected in Google Street View images. The models are trained to discover associations between cars and people. It is shown that the information about the "make, model, and year of all motor vehicles encountered in particular neighborhoods" has the potential to "accurately estimate income, race, education, and voting patterns at the zip code and precinct level".

Use in social science analysis: The built-environment attributes can be predictors of many socio-economic characteristics in neighborhoods. By making use of computer vision and deep learning, social scientists are able to infer unobserved socio-economic patterns based on observed built-environment features. This is different from the traditional way of investigating demographic data by surveying the population.

Explore with new data: it is possible to use satellite data for economic research. I am interested in performing deep learning data mining into satellite data to find potential insights into regional economic development.

[hesongrun]

Does finance benefit society? A language embedding approach Manish Jha, Hongyi Liu, Asaf Manela

Summary: This research measures popular sentiment toward finance using a BERT model trained on the five-word sentences containing 'finance' across eight languages from 1870 to 2009. The authors found persistent differences in financial sentiment across countries despite ample time-series variation. They found that sentiment toward finance declines after uninsurable disasters such as epidemics and earthquakes but rises following insurable risks such as droughts, floods, and landslides. Positive shocks to finance have a persistent effect on later economic growth.

Use in social science analysis: the methodology can also identify popular opinions toward other social science disciplines. With a better understanding of the driver of the sentiment, we may be able to find some deep underlying state that influences people's perception of the outside world.

Explore with new data: I am interested in GitHub data. It will be interesting to see different development communities' attitudes and sentiments toward certain advances in technology to better forecast future its future development.