search

langcog / experimentology

Experimentology textbook
https://langcog.github.io/experimentology/
Other
42 stars 18 forks source link

recommending difference score analysis in 9.1.4 #226

Closed poldrack closed 10 months ago

poldrack commented 11 months ago

In 9.1.4 you say the following:

The pre measurements can be used to subtract participant-level variability out and recover a more precise estimate of the treatment effect. ... This equation says “how much more did the treatment group go up than the control group?[13]

I know that this is common in psychology, but in biostatistics there have been strong critiques of the use of pre-post change scores in analyzing the effects of a treatment in randomized trials - e.g. see https://www.fharrell.com/post/errmed/#change or https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3286439/

I would reconsider the recommendation to use change scores, or at least add a note making clear that it makes some very strong assumptions that are unlikely to be met, and also mention that the use of the pre-test score as a regressor in the model (i.e. ancova) is generally recommended.

mcfrank commented 10 months ago

Hi Russ, thanks for continuing to engage!

The Bland and Altmann critique as I read it is mostly about difference between significant and not significant is not significant, right?

The Harrell points are good ones, but many of the assumptions are reasonable for psych pre-post designs and are things we talk about elsewhere, e.g. we strongly recommend not excluding based on the DV and we talk about the measurement issues that would invalidate pre-post subtraction (e.g., floor and ceiling effects, reliability of measure, etc).

I tried to change the footnote a little to make it clear that we're giving the subtraction as a conceptual illustration not a concrete analytic recommendation...