7. Deep Learning to Perform Causal Inference - fundamental

[lkcao]

Post questions here for this week's fundamental readings:

Grimmer, Justin, Molly Roberts, Brandon Stewart. 2022. Text as Data. Princeton University Press: Chapters 23, 24, 25, 26, 27 —“Prediction”, “Causal Inference”, “Text as Outcome”, “Text as Treatment”, “Text as Confounder”.

[XiaotongCui]

I have a general question about Chapter 24. I've observed that much of the research in Machine Learning (ML) and Causal Inference often emphasizes the high dimensionality of text data and the necessity to reduce its dimensionality to capture causality. However, I personally find it challenging to grasp the rationale behind this approach.

[Dededon]

The examples done by Egami and Fong show in the textbook are all experiment-based designs, using surveys with clearly defined questions toward recruited participants, and mined text treatments or outcomes. How can we combine the other survey datasets, like GSS and ANES, into our text analysis projects related with our own research interests?

[sborislo]

For Chapter 27, I don't understand how the use of lower dimensional text works for choosing covariates. Wouldn't a bunch of text falling under one topic increase the error term, thus making it more likely that the effect of truly confounding text is underestimated and, thus, less likely to be accounted for when estimating a causal effect? This can be seen in Figure 27.2.

[yuzhouw313]

When conducing causal inference on text data in chapter 24, Grimmer et al. introduced mapping functions to reduce raw text data to lower dimension summary/representation to fit into the causal analysis. While we have discussed multiple mapping strategies in previous weeks (LDA topic model and this week's Causally Sufficient Embeddings), I am curious to know more about in what ways can the mapping function account for confounding variables in text-based causal inference, and what are the limitations of these approaches, given the strict requirement for causality compared to classification or prediction?

[Twilight233333]

In discussing text as outcomes, examples in the book gather explanations for respondents answering yes or no. And extracted high-frequency words for each topic. Can this information be applied to other studies of deportation? Is it possible to judge an interviewee's preference for immigration repatriation based on his two statements?

[volt-1]

In ch.25,Text as an outcome can reflect the author's intentions, emotions, and perspectives, which are difficult to quantify. How can we effectively quantify and analyze the subjective elements in text, such as emotions and intentions, and integrate them with quantitative results?

[yunfeiavawang]

Similar to including covariates in regression models, we can include confounding of high-dimensional text variables in the causal inference model. I am concerned about the computational expense of this procedure. Though we can use the shrinkage method or penalize large weights, we still need to balance the feature complexity and model complexity. How could we find the balance point?

[michplunkett]

Causal inference, at least in my exposure, feels like it can very much be its own field of study. Based on the classes I've taken and papers I've read, it seems like you'd need someone to keep a constant watch on the experiment from an outside perspective to make sure any given study maintains its ability to make a causal statement. When bringing casual inference into large corpus studies, is it meant to be a more collaborative process or one where social science researcher are experts in causal inference as well?

[bucketteOfIvy]

In Chapter 24 Section 3.3, Grimmer, Roberts, and Stewart propose data splitting as an alternative to pre-registering analyses. This, to me, is a highly appealing alternative, as it allows for the main concern I have with preregistration (lack of ability to adjust to unforeseen circumstances) to be resolved while still ensuring that the insights gained from a study are actually there and not the result of too many researcher degrees of freedom. However, data splitting of this sort could also increase the variance in our categories, making phenomenon harder to detect in the first place. Does this tradeoff mean that preregistration is (in some senses) a "better" idea when dealing with small datasets, while data splitting is a better option when dealing with larger ones?

[QIXIN-ACT]

I still find myself somewhat puzzled by the difference between causal research and non-causal research. At times, it can be quite challenging to discern between the two. Furthermore, I'm curious about whether causal inference is always considered more valuable than non-causal inference, or if the significance varies depending on the context?

[donatellafelice]

I am curious at a high level about the ethical implications of social scientists refocusing on predictions through ML, especially given what we have learned about the inherently black box nature of many of the processes. what sort of additional verification need to be put in place when we're thinking, at this rather basic level, about forming predicative tasks? and what sort of verification should ethically be required for those trying to predict outcomes that could have really profound impacts on people's lives? is there industry standard IRB type review here? What about the ethics associated with social forecasting?

[naivetoad]

Can we develop standardized metrics or benchmarks for evaluating the accuracy of causal inferences drawn from text analysis?

[ethanjkoz]

In chapter 24, Grimer et al. discus a short introduction to causal inference. They introduce the concept of the stable unit treatment value assumption (STUVA). They explain the importance of this assumption in the text, but I can't quite wrap my head around this idea. Furthermore, I was wondering if the notation in their equations in 24.1 could be explained in a little more detail. (The one that looks kinda like this: ˉ​T​−yˉ​C​=pT​(yˉ​Ti∈T​−yˉ​Ci∈T​)−(1−pT​)(yˉ​Ti∈C​−yˉ​Ci∈C​). I was also curious if the other assumptions (ignorability and positivity) could be elaborated on a little more.

[h-karyn]

For Chapter 24, I didn't fully understand why overfitting is an important issue for causal inference with text. According to the book, a random word could be associated with the treatment by chance, not systematically. Intuitively, this issue is common regardless of the data type, and usually having a larger dataset can alleviate the problem. How is it special for text?

[beilrz]

I do not have much previous training on causal inference. I was wondering when would be causal inference be appropriate to use, instead of non-causal inference? For our textual data, should we have some prior assumption of causal relationship before running causal inference?

[alejandrosarria0296]

Given my limited exposure to causal inference I'm curious about the centrality that textual data should have in causal inference designs. When doing causal inference using text as data, should the causal inference design come first and then be supplemented by language processing strategies or should the text and its constraints take priority in the research design?

[cty20010831]

I think causal inference itself is a quite sophisticated topic, a topic I did not have much previous experience with. Hence, while reading Chapter 24, I had trouble understanding CAUSAL INFERENCE WITH g. How does this approach work? Are there any practical application of this method?

[Vindmn1234]

As language models continue to evolve, what advancements are needed to enhance their application in causal inference? Are there specific areas within NLP or causal inference research that should be prioritized to improve the accuracy and interpretability of causal estimates derived from text data?

[joylin0209]

Is it possible to establish standardized metrics or benchmarks for evaluating the accuracy of causal inferences drawn from text analysis? What considerations should be taken into account when developing such metrics, considering the inherent complexity and subjectivity of textual data?

[chenyt16]

When conducting causal analysis using numerical data, we first need to determine if the influence of the independent variable on the dependent variable is significant. For example, in a regression model, we assess this through the p-value of the coefficient. However, when using text data, this process seems to be omitted.

[ana-yurt]

In chapter 26 "Text as Treatment," the authors mention the strong assumptions needed to identify casual effects—that other features are independent of the treatment feature or have no causal effects. In real life, how do we make sure our data satisfy those somewhat difficult assumptions? What are ways that we can test those assumptions, and when should we not make them?

[runlinw0525]

How does combining sequential experiments with train/test splits, such as those used in immigration and presidential speech studies, improve the reliability and applicability of our conclusions from textual data across different social science research areas?

[erikaz1]

How many randomizations do we need to determine and validate the best codebook function g? How should our approach for finding g change if we have a corpus with diverse documents, compared to a more linguistically/stylistically/contexually homogenous corpus?

[anzhichen1999]

How do we mitigate the risk of bias in the final estimation step of the STM, where topic proportions of all control documents are reestimated as though they were treated, to ensure that the comparison between treatment and control groups remains valid and reliable?

[HamsterradYC]

Given the challenges posed by cross-validation in accurately forecasting future data, particularly amidst systemic shifts, and acknowledging the impracticality of solely relying on real-time predictions for model adjustments in certain research domains, what methodologies or approaches might be adopted to enhance and maintain the predictive precision of models over time?

[yueqil2]

In these chapters, one of the critical step to use text in causal analyses is reducing dimension. However, what's the different between low-dimensional textual data and other types of variable like numeric? If we just label the texts with numeric scale and then do the regression, I don't think it's fair enough to regard it as a causal inference based on text data.

[Brian-W00]

How can the methodologies discussed for 'Text as Treatment' and 'Text as Confounder' be applied to emerging technologies like AI-generated texts, where the authenticity and origin of text data might blur traditional boundaries of causal analysis?

[icarlous]

In exploring causal inference with limited exposure, I'm curious about the role of textual data. When using text for causal inference, should the design precede and be complemented by language processing, or should the focus be on prioritizing the text and its constraints in the research design?

[YucanLei]

I find it difficult to quantify and analyze the intention of the texts, for example, how do we know if somebody is been sarcastic? Sometimes, it is difficult to conclude the intention from texts alone, particularly if the speaker is not trying to make it obvious.

[Marugannwg]

I understand the fact that we need to reduce the dimension of the text embedding to enable the process of causal inference. I'm not very familiar with the underlying math and mechanism, but I can envision the process; however, I found it unsettled if I'd need to persuade the effectiveness and reliability of this process. What is the assumption of casual inferences (especially, consider we are working with some preprocessed text data)? Also, how to illustrate, discuss, and validate the outcome?

[Caojie2001]

The reduction of dimension for word vectors seems to be a very common preprocessing operation before model construction. However, from my intuition, dimension reduction may also lead to loss of information. I wonder how we should balance the pros and cons of dimension reduction.

[Carolineyx]

It is interesting to learn how to predict and draw inferences from text data. I'm curious about how we could clear/lower the noise/errors in prediction, especially if we're not sure what the confounding variable is.

[JessicaCaishanghai]

Text as data has become a very interesting field where further exploration is needed to help the research forward. However, I'm interested in what's behind the causal inference. For example, we already know that length of the post is closely related to the emotions conveyed, but what's the underlying mechanism? I feel like we still need expertise in psychology or other behavioral science to further investigate the ultimate reason logic.

[floriatea]

Given the capabilities and examples of using text for predictions in the document, what are the limitations in the granularity of predictions that can be achieved? For instance, how detailed and specific can predictions based on text data become, and what factors limit the precision of these predictions in practical applications, such as predicting economic trends or public health crises?