lecy
commented
3 years ago There’s not a perfect solution for this. We always worry when important details are omitted. On the other hand you can’t include every detail about a study so there is actually a lot of skill in writing a Goldilocks research design overview.
I would go by the rule of thumb selection and attrition are guilty until proven innocent, and the rest need some evidence that the problem exist (we first assume innocence). Or in the regression to mean case knowing the sampling frame would generate the issue (sampling from tails), even if authors don’t mention it.
Your brain can always invent scenarios in which something is true (one of the reasons humans are bad at causal reasoning - post-hoc sensemaking). Doesn’t mean it is really an issue. Look for clues and make your case.
In the instructions for the Campbell Scores on Lab 03, for several items they say "In order for this item to get a zero, you must first argue that it exists and then show that a lack of adequate controls will lead to incorrect results." (Maturation, Secular Trends, Seasonality) Can we give it a zero if we think that issue might exist and the study does not address the issue? For example, if a study measures healthy eating over time and the article doesn't tell us what time of year the study took place, it's conceivable that subjects' healthy eating behavior could be affected by holidays (seasonality). Would that possibility (if not addressed in the article) be enough to give the item a zero? Or do we have to have evidence that the issue positively exists in order to give a zero? Thank you!