Week 8 - Possibility Readings

[bhargavvader]

Post a reading of your own that uses deep learning for social science analysis and understanding, with a focus on strategic sampling and active learning. Here, you may also choose examples that built multiple "worlds" using a deep learning model and compared them (e.g using a word embedding model trained on two different datasets).

[Yilun0221]

Title: Data sets: Word embeddings learned from tweets and general data

Summary: In this study, the researchers studied how word embeddings work in exploring Tweet information. The researchers created ten word embedding datasets from the tweet corpus. They also used “general data” in the research, which consists of “news data, Wikipedia data and other web data”. The researchers also created word embedding data sets using the general corpus and the corpus consisting of the general corpus and tweet corpus. There are three methods employed to create word embeddings, including “distributed word representation, phrase identification and tweet spam filtering”. After getting the word embedding representations, the researchers used them to do perform sentiment analysis and topic classification tasks on the tweet corpus. For the sentiment analysis, the tweets would be labeled as negative or positive. For the topic classification task, the tweets will be classified into 11 categories, including “Sports, Politics and Business”. Generally, the word embeddings achieved good performance on both tasks, where the comprehensive corpus that includes the general data and tweet data seems to give better word embeddings. The researchers agreed on the power of distributed word representations and phrase detection.

Expansions to social science analysis: I think most corpus in real life have the same characteristics as the researchers described (in terms of tweet data) in the paper: “Tweets are short, noisy and have unique lexical and semantic features that are different from other types of text.” Many corpus datasets are not well cleaned when they are created in social science (like people’s reviews/comments), so I believe this paper can enlighten us to how to deal with this kind of data.

New dataset exploration: Corpus from other social media platforms can be explored to help policymakers understand how citizens think of a social issue/a policy. Also, I think we can also explore articles of a newspaper/media (like BBC) to analyze their views on certain topics.

[Raychanan]

Title Active deep learning method for semi-supervisedsentiment classification

Summary In natural language processing community, sentiment classification based on insufficient labeled data is awell-known challenging problem. In this paper, a novel semi-supervised learning algorithm called activedeep network (ADN) is proposed to address this problem. Experiments on five sentiment classification datasets show that ADN and IADN outperform classical semi-supervised learning algorithms, and deep learning techniques applied forsentiment classification.

Expansions to social science analysis This paper proposes a novel semi-supervised learning algorithmADN to address the sentiment classification problem with a smallnumber of labeled reviews. ADN can choose the proper trainingreviews to be labeled manually, and fully exploit the embeddinginformation from the large amount of unlabeled reviews to improvethe robustness of the classifier. This method is readily available and helpful to social science researchers. With the help of this algorithm, researchers can avoid spending a lot of money and save on inviting human beings to label a large amount of data. Instead, they should only label a very small number of samples. to make the sentiment classification efficient.

Dataset This technique should be applied to countless datasets. For me, it's worth implementing this method on my research quesiton of sentiment changes on microblog during COVID-19.

[william-wei-zhu]

Title: Human Trajectory Forecasting in Crowds: A Deep Learning Perspective

Summary: this paper applies deep learning to conduct human trajectory forecasting, which means when pedestrian walk toward each other on public streets, the researchers want to predict the direction and speed of their following movements. The researchers group pedestrian interactions in categories including "leader-follower", "interacting", "collision avoidance", etc. The researchers also introduced a platform called TrajNet++ to evaluate the performance of different human trajectory forecasting models.

Expansion to social science analysis: the researchers point out that social norms regulate how pedestrians will react when other people walk toward them on busy streets. It will be interesting to see how pedestrian interactions differ when social norm changes. I am particularly interested in seeing how the model will adapt from pre-covid to post-covid.

Data: videos of busy streets before, during, and after the covid-19 pandemic, group by streets in different countries.

[cytwill]

Title: ActiveHARNet: Towards On-Device Deep Bayesian Active Learning for Human Activity Recognition

Summary: The authors proposed an interesting work of human activity detection from mobile devices via deep Bayesian neural networks, active learning, and incremental learning. The deep Bayesian neural network is used not only to make final predictions but also to provide the posterior uncertainties of results from the dropout layers. For the active learning process (deciding the next query data point), they tried four different acquisition functions to find the most suitable one:

• Max Entropy: pool points are chosen that maximize the predictive entropy.
• Bayesian Active Learning by Disagreement: pool points are chosen that maximize the mutual information between predictions and model posterior.
• Variation Ratio: The Least Confident method for uncertainty-based pool sampling is performed.
• Random Sampling: pool points are picked uniformly.

The pipeline was tested with two wrist-worn datasets: the HHAR smartwatch dataset and the Notch Wrist-worn dataset, and real users are incorporated as label annotators for selected query data points. Also, to simulate the situation where new data would come from the sensors as streams, researchers experimented with incremental active learning with LOOCV on their datasets, which allows them to continuously train the model with incoming data. Their results suggest that using the variation ratio as the acquisition function yields the best result for both datasets and different users.

Extension to Social Research: While this research does not directly explore any social science questions, it provides great potential for social science researchers to efficiently use observational data from these mobile devices. The average accuracy of inference achieved in this paper is desirable (around 90%) considering its relatively low requirements for labeled data. The behavior data collected can be used in lots of social research like people’s change in lifestyle due to certain treatments (for example, stress level or social accessibility, the pandemic is an example of low social accessibility). Moreover, considering expanding this technique to Web of Things or IoT, we might even be able to capture more reliably categorized human-centered data, which would provide more possibilities for social science studies.

New dataset exploration: Though the datasets used in this study are existing mobile-device datasets, I think the best place to implement this pipeline is some real mobile devices in use, like Apple Watch, so we can obtain how this model works on up-to-date real data. Also, in some special scenarios where people’s behavior data would be recorded like VR rooms or under some IoT settings, there could also be chances to apply the pipeline to identifying some pre-defined behaviors from these scenarios for specific research questions.

[k-partha]

Deep Bayesian Active Learning with Image Data - Machine Learning Research

Summary: The authors develop a novel Bayesian CNN approach to improve upon existing active learning methods for high dimensional problems - particularly focussed on images in this paper. Active learning for high dimensional problems has proven to be highly difficult to develop - this paper is one of the few (at the time of publishing) to pursue an active learning approach to high dimensional data. They demonstrate their technique on medical imaging data and show that their technique has the potential to save human labellers time and also improve attempts on the problem using active learning.

Extension to social science research: The development of general-purpose active learning approaches to image classification would have very important implications for social science as it has the capability to reduce costs significantly. Researchers can pick out the particular profiles that they would have to obtain further data on and target the relevant sub-population. This paper takes a step in that direction.

New dataset exploration: This could be applied to problems like the personality prediction problem from Twitter data that I'm working on to great effect. Instead of limiting my choices of profile pictures to the ones that have self-declared prior to the start of data collection, we could follow up on particular profiles within the entire constellation of the Twitter universe to efficiently build a dataset to build a deep learning model that classifies personality traits from profile images. i.e I could selectively survey (for personality details) the profiles that the model recommends that I obtain further data.

[pcuppernull]

Title: Deep Active Learning by Model Interpretability

Summary: This paper addresses the challenge of sample selection in active learning. The authors propose a methodology called DAMI to improve active learning sample selection. Intuitively, DAMI leverages the notion that nonlinear neural networks that employ piece-wise linear activation functions (for example, ReLU) are mathematically equivalent to a set of local linear classifiers. DAMI uses the weights of the linear classifiers to linearly divide and cluster samples. These samples of unlabeled data are then proposed to be labeled and included as part of the training data for a model. The authors test this methodology on a variety of tabular datasets and demonstrate performance that is consistently superior to alternative methods.

Social science extension: DAMI could be particularly useful for social scientists. Generally, it hinges on the interpretability of deep neural networks, which could help assuage some concerns that deep networks remain “black boxes”. I would be interested in extending this methodology to text data – many social science applications use large text corpora that require some degree of hand labeling. DAMI might be effective in helping researchers streamline the hand-coding of training data, which could accelerate research and make it more cost effective.

Dataset: I am interested in exploring the role of technology companies in politics. A researcher could collect a corpus of geo-located tweets that contain the word “Facebook” and hand code a selection of the tweets as exhibiting (or not) distrust towards Facebook. DAMI could be applied to help the researcher identify subsets of tweets to hand code in order to efficiently improve the accuracy of the model. These classifications could be compared against vote outcome in U.S. states or counties to examine the connection between mistrust of Facebook (or other technology companies) and voting behavior.

[hesongrun]

Title: Active Learning for Convolutional Neural Networks: A Core-Set Approach (ICLR 2018) by Ozan Sener, Silvio Savarese https://arxiv.org/abs/1708.00489

Summary CNN has been successful in domains with a large dataset of supervised examples. However, the approach is restrictive when collecting a large number of labeled images is very expensive. One smart way to solve the issue is to come up with smart ways to choose images to be labeled from a large collection, i.e. active learning. The authors' empirical study finds that many active learning heuristics in the literature are not effective when applied to the case of CNNs. The authors defined the problem of active learning as core-set selection, i.e. choosing a set of points such that a model learned over the selected subset is competitive for the remaining data points. Using this core-set selection technique, the model outperforms existing approaches in image classification experiments by a large margin.

Social Science Extension There are many image labeling tasks in social science studies. For example, to study social media sentiment transmission, one needs emotion labels for different images people post on social media. Furthermore, in the economics domain, satellite images can be obtained easily but the local living conditions may take great survey resources or efforts to obtain. In these areas, this core-set active learning algorithm posits great potential for research opportunities. Researchers can obtain important labels using this method.

Dataset Potential unlabeled social science image dataset, e.g. the Twitter image dataset, dataset of satellite images and etc.

[nwrim]

Sonlik et al. (2017). Bayesian Optimization for a Better Dessert, Proceedings of the 2017 NeurIPS Workshop on Bayesian Optimization * (NeurIPS was called NIPS back then)

1. Brief summary of the article This paper demonstrates a real-world use case of using Bayesian Optimization (BO) to sample the parameter space. Basically, Google wanted to make better chocolate chip cookies using BO. So they made some cookies (initial batch), fed some Google employees ("human experts") and got some ratings back, fed the BO with the rating result. Then the algorithm gave a new cookie recipe which was rated again. The last process was repeated until Google had good enough cookies. So basically they were sampling the parameter space (recipes in this case; e.g., how much sugar should be in the cookie) using the BO algorithm, and their evaluation metric was done by human raters. Some points that were interesting to me was:

   • The embedded the "domain expertise" in the recipe - the Google food service team sometimes were able to adjust the recipe that the algorithm suggested (although they modified it less and less as the experiment went on - the workers had more beliefs on the algorithm's suggestion)
   • The final cookie ended up being quite different from "conventional" recipes, violating the expectation of experts in some sense
   • The final cookies showed a huge difference based on where it was tested out (PA or CA). I think this shows a fun case where strategically sampling the parameter space can lead to interesting findings, with a little flavor of a human experiment!

2. Suggestion on how its method could be used to extend social science analysis We social scientists usually care about the underlying function of the phenomenon (f(x)), such as the coefficient of the variable in the linear regression. However, this function is really hard to grasp if we are in a high-dimensional complex space (black box models can grasp it, but it often does not give us intuition about the function) and sometimes just having the argmax is enough (so argmax f(x) instead of f(x)). In that case, I think sampling parameters via BO makes getting the argmax much more efficient than typical, purely random sampling. So basically, I think this kind of approach will be useful in social sciences when (1) we care about argmax, rather than the function itself and (2) the sampling space is very complex and high-dimensional.

3. Describing what social data you would use to pilot such a use The experiment we talked about in class briefly (and some will get some slides from me later today or tomorrow) could hopefully be a good pilot. Just to summarize it briefly here, James and I are trying to use BO to sample the team characteristic parameters efficiently to build a team that works very well on certain tasks.

[jsoll1]

Deep active learning for object detection SOUMYA, ASIM, VINAY

Summary: This paper uses the human in the loop system of active learning to develop a more efficient object detector method. It's really efficient: they reach incredibly high levels of accuracy (>95% of that of normal supervised learning) with just a third of the training data. Their technique is to randomly select a subset of images that aren't annotated, have them be annotated by some arbitrary power and then train the SSD on the annotated images. They keep repeating the process of selecting images to annotate and retraining until some threshold is reached. the active learning component is a white box method that uses query by committee to choose the objects to annotate based on the architecture of the video.

Social Science Extension: This isn't directly connected to social science applications (though I will describe how these techniques could be used for some). It's important to me because deep learning based object detection algorithms may be a component of my final project for this class (using them as they relate to traffic to measure congestion). However, I do think that there are a lot of times in social science when it's important to be able to detect objects (such as when examining war images in political science contexts).

Which Dataset I'd use: I'd use one of the traffic datasets that William and I have been going through. Probably google streetmaps data.

[luxin-tian]

Deep active learning for named entity recognition Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, Animashree Anandkumar

Summary: Typical NLP tasks including name entity recognition (NER) usually requires large amounts of labeled data. In this paper, the authors demonstrate that the amount of labeled training data can be drastically reduced when active learning is combined with. The authors introduce a lightweight architecture for NER, which is computationally much more efficient than other best performing models, and their model achieves nearly state-of-the-art performance on standard datasets.

Social Science Extension: much social science research that is combined with data mining, especially with NLP requires NER as a pre-processing procedure. This algorithm can be used for training NLP language models and tuning NER models on special corpora, for example, archeological or historical context, on which models trained on modern language can perform worse. Active learning can help reduce the workloads of manual labeling.

Dataset: Google Books.

[ajahn6]

Adversarial Active Learning based Heterogeneous Graph Neural Network for Fake News Detection, Yuxiang Ren, Bo Wang, Jiawei Zhang and Yi Chang

Summary: Fake news presents challenges for detection including its intention to camoflage itself as legitimate and its potentially low volume in relation to legitimate news. An effective fake news detector must be able to handle the paucity of labeled data, heterogeneity of fake news, and generalizability. Treating this problem as a problem of node classification for heterogeneous graphs, this paper proposed a Adversarial Active Learning-based Heterogeneous GNN. This model takes a GNN with some labels and uses a Hierarchical Graph Attention (HGAT) classifier to label more nodes. These are then fed into an adversarial selector vs expert labeling. This combines the effectiveness of GANs on heterogeneous graphs with Active Learning's ability to work with more sparsely labeled data. The paper found that their AA-HGNN model outperformed models based on text classification or graphical models with comparably limited labels for training. This is especially advantageous to fake news because of the imperative to detect it in its early stages before it spreads, when there is likely to be limited labeled data for training models.

Social Science Extension: This paper was written with social science in mind, namely the problem of misinformation propagation across social media. The authors emphasize the generalizability of their model to other graphical applications where labels might be sparse. This could include any manor of online content that is produced or propagates at such a clip that taking the time to build a robust labeled dataset would be a limiting factor.

Pilot project: One potential project might be the classification of extremist or TOS-violating memes on a site such as Instagram or image board like reddit. A graphical map representing likes, posts, or comments from certain users and a relatively small subset of labeled data could be used to train the model for finding other such memes that have not been labeled. This could be especially useful for meme image classification which can often be obscure or esoteric and hard for an image-based model to tease apart, since the AA-HGNN does not need to take content into account at all, only the graphical network representation.

[bakerwho]

Taking a bit of a liberty in translating from active learning to 'interactive learning', this is a really cool paper!

A Review of User Interface Design for Interactive Machine Learning John J. Dudley and Per Ola Kristensson. ACM Transactions on Interactive Intelligent Systems. 2018.

Summary: This paper describes the interactive workflow of IML (interactive machine learning) as inherently co-adaptive - the user and the model must directly influence each other’s behaviour. It nods to complexity theory in offering frameworks to deal with emergent behaviors, edge cases, and more non-linear applications of a system that is more than the sum of its parts. It also offers a framework based on transforming data inspection and correction into a process of heuristic-driven and goal-driven exchange between human and machine. While it offers a useful framework, it acknowledges that work in this field is nascent and offers many interesting ideas for future work.

Social science extension: It would be interesting to decouple the loop of human beings reacting to advertisements or other 'nudges' by attempting to collect feedback on how the nudges affect future choices, as well as the human's reaction to the nudge in the first place. This zone of conflict (where the algorithm attempts to influence the behavior of the human, but is also invested in maintaining the human's sense of autonomy or personal agency) seems crucial to design better human-computer interactions in general, as well as aligning machine behavior with human priorities in a decentralized way.

Dataset: I can only envision an MTurk-based or similar study being sufficient to explore the effects of interactive design elements on machine learning with humans in the loop. Trying to think through other contexts in which this can be studied is an open problem!