Feedback on Chapter 1 - Githubissues

Major feedback

Why correlation doesn’t equal causation It seems odd to me that the only limitation of observational studies we discuss is confounding. To strengthen the argument for experimentation, we may want to discuss that experiments help us distinguish between three patterns that can lead to a correlation between A and B:

A causes B B causes A C causes both A and B to co-occur

If we wanted to work this in, we might have to move away from the “snipping analogy”. For example, we can say that manipulating the variable allows us to establish temporal precedence. Because we manipulated A and then measured B, we can make conclusions about whether A caused B.

Where to talk about confounds? Currently, we discuss confounds in the context of why we manipulate things. I think this makes sense, but it feel it would be better placed after the discussion of “method of differences”.

For example, we can say “In an experiment we would want to make sure that the only thing that we change is whether people listen to Dylan. Thus, if we are concerned about age, we would make sure to keep it constant. We might make sure that everybody listens to Dylan regardless of age. Or we may make sure that the group that listens to Dylan is a similar age as the group that doesn’t. In doing so, we get to address the age confound.

From here, we could transition to discussion of randomization by acknowledging that it’s hard to control for everything. To drive this home, we could give a few example: mood might be related to writing ability, so we want our groups to be in roughly the same mood. Time of day may be related to writing ability, so we want our groups to be run at roughly the same time. There is seemingly an infinite number of potential confounds. How can we control for all of them? By using randomization!

Generalizability Ethics Box If you’re looking for a more psych-related example, you can talk about this paper: Culture Matters When Designing a Successful Happiness-Increasing Activity: A Comparison of the United States and South Korea

Summary: psychologist were promoting gratitude as a well-being intervention based largely on data collected with WEIRD samples. When they ran the study in South Korea, though, they found that the gratitude intervention decreased well-being. They posited that this might be occurring because the gratitude intervention in South Korea is leading to mixed feelings of not just gratitude—but also indebtedness.

Exercise 1.1. I’m not familiar with the phrase “condition structure”. Is this essentially asking them “how would you design the intervention”?

The reader may not know what “nuisance variables” mean. We mention that phrase in a footnote, but call it a “confound” in the main text.

I worry that question B will be too difficult for the reader. Maybe instead it would be useful to ask them to reflect on potential generalizability constraints (e.g., if their experiment might yield different results with different samples, different manipulations, and different dependent variables)

Minor feedback

“The edge between them represents a hypothesized causal relationship.” It seems more intuitive to describe it as the “arrow” between them.

“Perhaps this is because experiments are a critical method for making strong causal inferences, which are otherwise in short supply in psychology, and so the contrast between experimental and non-experimental research is very salient for psychologists (and economists too).” Flagging this as a long sentence, which I feel are tiring to read. I recommend adding a period after ‘psychology’ and starting a new sentence with “thus”.

For the observational study, I would make things more concrete I know we are fine with people Googling things, but the phrase “observational study” doesn’t add anything and may be confusing for an advanced undergraduate. A potential rewrite: “Returning to the Dylan Hypothesis, suppose we didn’t run an experiment, but instead merely asked people how much they listen to Dylan and tested how well they write.”

After this sentence, we ask the reader to imagine what the study might yield. However, I think it would be more straightforward to ask them to focus on a hypothetical result. For example: “Imagine we found that people who listen to Dylan tended to be stronger writers (i.e., that Dylan listening was positively correlated with writing ability). Can we make a causal inference?”

Footnote 5 I don’t think discussing the backdoor criteria adds much. The phrase doesn’t make a lot of sense to people who are not familiar with DAGs.

In the language of graphical models, if we control the Dylan listening, variable is causally exogenous – not caused by anything else in the system). Extra “)”

Randomization – whether of the order of an intervention in a sequence or the assignment of participants to conditions – is the key intervention that is guaranteed to break causal dependencies. I suggest removing the word “guaranteed”. We immediately add some nuance that makes it clear that there are not many things guaranteed in the life of the experimentologist.

“fundamental” psychological like visual perception, I think this sentence is missing a word. Is it supposed to say “fundamental psychological process”?

langcog / experimentology

Feedback on Chapter 1 #60

Major feedback

Minor feedback