7. Deep Learning to Perform Causal Inference - orienting

[lkcao]

Post questions here for this week's oritenting readings:

Veitch, Victor, Dhanya Sridhar & David M. Blei. 2020. “Adapting Text Embeddings for Causal Inference.” Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), PMLR volume 124, 2020.

[XiaotongCui]

I'd like to pose a question regarding the essence of ML for causal impact. The main idea is to derive treatment probabilities through various model training, essentially creating a mapping from embeddings to expected outcomes for both treated and untreated instances. However, during the training process, I'm unsure how the texts used for training obtain their true values for this information. Do researchers generate the texts themselves for this purpose?

[Dededon]

The architecture provided by the paper is very inspiring: they modify the BERT model's architecture to let the LLM to learn on the treatment effect. My question for the causal BERT model is that this model has limited migratability. Preparing the data Fitting the new BERT model may need lot of coding knowledge for general social science audience. For causal inference tasks, do we really need to use this complex architecture? In most causal infrence cases, do we really need to evaluate the causal effect of every word token, or the BOW vector of the whole document? Aren't topic modelling, or simple NER information retrieval good enough to finish most independent variable construction?

[sborislo]

I don't understand how the authors' approach allows for "sufficient" causal inference in natural examples. The portion that raised this concern the most for me was the fact that the authors supervised the models to allow them to reduce confounding. This seems much more difficult to do in non-synthetic cases, and the authors seem to suggest that, even this, only reduces confounding. This seems analogous to controlling for covariates in a regression, which does not establish causality.

Why do the authors think their approach virtually eliminates confounding such that causality can be claimed?

[Twilight233333]

The causal inference proposed by the author works well for variables that control text expression (e.g., not just male labels, but also the way men speak can affect popularity), but for other causal analyses that do not require tone control, such as I want to analyze the effect of different emotions on other real-world metrics such as clicks. Is textual causal inference still effective in this case? Or is it better if we just do a sexiness analysis first to create label and then use a purely empirical model?

[yuzhouw313]

The concept of reducing text dimensionality to efficiently preserve essential information while eliminating less relevant details is both efficient and logical. The authors' strategy to retain just enough information for predicting treatment effects and outcomes seems robust. However, the explanation of implementing causally sufficient embeddings raises some questions, particularly regarding the use of a "black-box" tool for distilling words into information pertinent to prediction problems and solving those problems (p. 3). How can we ascertain that the embeddings λ(w) are causally sufficient, especially considering the opaque nature of the black-box model? Specifically, how do we verify that the model's structure allows for accurately estimating propensities and conditional outcomes required by the downstream effect estimator (p.3)?

[volt-1]

The potential of using text embeddings for causal inference is innovative. I am interested in the generalizability of their approach, especially when applied to different types of text data, such as social media posts versus academic papers. Does the effectiveness remain significant across these varied text types?

[michplunkett]

First, the black box nature of the embedding methods makes it difficult for practitioners to asses whether the causal assumptions hold...

The concept of using text embeddings for causal inference is really fascinating, and I imagine the applications of it extend beyond what feels immediately imaginable. Along with @volt-1's concerns about the extendability of the methods, I have some concerns about the general opaque nature of the process. The paper seems like it'd incredibly opaque to anyone, I assume, that does not work at the convergence of ML and graduate level statistics. Is there any concern about the overly complex methodology of this paper, or is that something that someone should expect when working at the 'bleeding edge' of new disciplines?

[yunfeiavawang]

This awesome paper shows the potential of adapting text embeddings for causal inference with the scientific paper example and gender on Reddit example. The causally sufficient embeddings combine two ideas. The first one is supervised dimensionality reduction and the second is disposing of linguistically irrelevant information, which for me are procedures filtering out the unimportant confounders in determining the outcome variables. I am not sure if they could guarantee the causal relationship between theorem and paper acceptance, and between gender expression and post popularity. My understanding is that they are only more accurate investigations of the correlations but not a validation of the causal effect.

[bucketteOfIvy]

Building a bit on @volt-1 and @michplunkett 's concerns, this method (to me) seems relatively reliable and powerful, but also like the sort of thing that one will have to convince their colleagues isn't a black box. Are there other common casual inference from text methods which one could use as a "sanity check" to convince skeptics that this model is doing what it claims to be doing? If so, what are some of the common and useful options?

[Marugannwg]

Seems like what the authors attempted was to separate some key information from the utterly rich and dense text material -- they (assumed?) that the large corpora would definitely include all the information that indicates what confounds people's judgment of the paper. I really need some help to connect the embedding with those concepts confounding. Literally, what is the essential part of this (almost black-box-like) method can identify the confounding aspect of the research questions?

[ethanjkoz]

This was a very inspiring and imaginative paper for thinking about how to apply causal inference to textual data, a task I did not know was possible. One key assumption of this paper is that raw textual data contains lower dimensional properties meaningful/sufficient for causal inference. However, within the context of posts on Reddit, how reliable is this method for posts of small sizes (let's say less than 25 words or so). How much noise do small posts add to the data and muddy causal performance? Furthermore, how reliable is this classification for different documents by the same author? On another note, I am also somewhat confused by the interpretation of theorem 3.1. How do we know we have eliminated outside confounders? (I was thinking variables like platform or time of day, particularly with regards to the paper on troll posts).

[QIXIN-LIN]

"Causally sufficient embeddings" appear to be an intriguing method. Is there a specific threshold for the text that must be met to conduct the analysis? Additionally, for text data that does not meet this criterion, are there alternative methods available to perform causal inference?

[anzhichen1999]

In the paper, the author asserted that the validity of causal inference procedure partly relies on the embedding method's ability to extract semantically meaningful information relevant to both treatment and outcome, supported by the strong performance of these models in natural language tasks. However, how does it address the potential limitations or biases inherent in the underlying language models, such as BERT, which might influence the extraction of semantically meaningful information? For example, BERT's pre-training on specific corpora could introduce biases or limitations in capturing the nuances and contexts necessary for accurate causal inference in different domains. How does theapproach mitigate such risks to ensure the robustness and generalizability of causal inference conclusions?

[naivetoad]

How do causally sufficient embeddings compare with traditional methods for causal inference in terms of scalability and accuracy? Given that traditional methods might not efficiently handle high-dimensional text data, understanding the scalability and accuracy trade-offs could be insightful.

[donatellafelice]

I am wondering what actually would constitute as casually sufficient? In the paper they consider the presence or absence of something specific that will only occur once per document (gender label), or may occur more than once (theorem) but is a very specific feature that can be extracted. Could this be applied to a rate questions - so could you look with causally sufficient embedding to find out if the prevalence of a certain type of question in a conversation was predicative? IE asking does 3 or more open or closed question predict XYZ outcome?

[alejandrosarria0296]

From my understanding, the method explained on the paper hinges on the premise that the distilled embeddings retain enough information to accurately model causal relationships between treatment and outcome. Given this interplay of dimensionality reduction and causal inference, how does the proposed method ensure that the process of embedding and dimensionality reduction does not inadvertently omit critical information that could affect the causal relationships being studied? Moreover, in practical applications, how does one balance the trade-off between dimensionality reduction for computational efficiency and the preservation of causal nuances within the text data?

[muhua-h]

How do the authors ensure that the causally sufficient embeddings only retain information predictive of both the treatment and the outcome, without capturing noise or irrelevant data?

[runlinw0525]

This is a very fascinating paper. My question is that when estimating causal effects from observational text data, how can we overcome the challenge of high-dimensional text data? For example, how can we accurately infer causality from the subject of a scientific paper or the writing quality of a Reddit comment?

[cty20010831]

This paper highlights a novel way to utilize text to do causal analysis. My question would be how do causally sufficient embeddings compare with traditional text embeddings in terms of computational efficiency and accuracy?

[Vindmn1234]

This paper represents a significant advancement in the field of causal inference from text data, introducing an innovative approach to address the challenge of confounding variables through the development of causally sufficient embeddings. My question is: given the reliance on pre-trained language models like BERT, which may encode societal biases present in the training data, how do the authors address potential biases in causally sufficient embeddings? Are there specific measures or adjustments made to ensure that causal inferences drawn from these embeddings do not inadvertently perpetuate or amplify biases?

[joylin0209]

I find application examples where cause and effect are sufficiently embedded in real-life scenarios to be very enlightening, demonstrating their practical value. However, I am curious about the scalability of these methods when dealing with large-scale data sets across different domains.

[HamsterradYC]

In the paper, 'causally sufficient embeddings' are created to preserve parts of text data that are relevant to treatment and outcome predictions. This method is based on the assumption that there is enough information within the text to support causal adjustment. How can one effectively identify and retain information directly relevant to causal inference while compressing text data into a low-dimensional representation? Especially when dealing with text data that has highly complex internal structures and subtle differences, can the effectiveness of this method be maintained? Furthermore, considering that the data in the paper is limited to computer science papers and specific Reddit subforums (such as keto, okcupid, and childfree), I am curious about whether this method has sufficient generality to be extended and applied to other fields.

[Caojie2001]

This article proposed a feasible approach to conduct causal inference on text data using embedding methods, which is very appealing to me, as I have always had an intuition about ML methods and research that the process of reasoning from data, analyses to conclusions is rather ambiguous. However, I still wonder whether the application of this method could be extended to other types of corpora and projects, or we have to develop different working processes to deal with them.

[yueqil2]

This article seems to deploy a bunch of knowledge we've learned so far to study causal effect. Is there any other possible combination of what we've learned to design a causal study that works better than this?

[chenyt16]

In this paper, the authors use BERT as the backbone to train the model. Given the development of language models in recent years, could this method be applied to other language models, such as llama, to achieve better performance?

[erikaz1]

Why is it crucial that the simulated text used for validation involve both treatment and confounder?

[beilrz]

I think one major issues, as mentioned by the authors, is the lack of high quality data set for causal effect directly. As causal effect could be difficult to observe, I was wondering, in our research, how could we test the accuracy/precision of these models? since we may not have a labeled test set.

[Brian-W00]

Regarding the topic discussed in the paper, what are the most recent advancements or studies that further explore or challenge the findings presented, and how do they contribute to the field?

[Carolineyx]

it is interesting that BERT can be used in many ways. I'm wondering how we can verify the result, is there another technique that we can replicate the result so that we know the results can be trusted?

[icarlous]

This introduction of the word embedding to casual inference is impressive. My question is how to compare different kinds of embedding like word2vec, transformer, or from Bert? How to qualify them compared to statistic embedding like tf-idf.

[YucanLei]

It seems like the author assumed the paper is free of biases and inaccuracies. From what I have done in the project, I believe it is safe to say a good chunk of online posts are nothing beyond noises. They are not good for causal inferences.

[ana-yurt]

The authors of the paper assume that the text fully captures confounding. I wonder if this assumption is fully sound for the measures of popularity on Reddit? For example, won't popularity partly reflect the societal attention directed by news cycles?

[JessicaCaishanghai]

Causal inference combined with the language models and text is very interesting and useful, and I can immediately think of many different scenarios to implement the causal inference. However, the paper has made some default setting and I think causal inference indeed is harder to capture than our expectation.

[floriatea]

The paper discusses adapting language models for causal inference within specific contexts (e.g., academic paper acceptance, Reddit post popularity). How might the approach detailed in the paper be adapted or expanded to generalize across a wider range of domains, including sensitive areas like healthcare, legal judgments, or financial decisions, where text data plays a critical role in decision-making processes? What potential exists for causally sufficient embeddings to be applied in real-time language processing systems, such as automated content moderation or personalized recommendation engines, to better understand the causal impacts of language on user behavior?