Replication Crisis - Reading & discussion

[DefenderOfBasic]

https://www.mod171.com/p/replication-crisis-the-syllabus

Wednesday, 7 September

Andrew Gelman – What has happened down here is the winds have changed

• very interesting that Susan Fiske is arguing that maybe more openness is a BAD thing? "it can derail & threaten process". "online vigilantes" "data police"

In short, Fiske doesn’t like when people use social media to publish negative comments on published research

the research incumbency rule: that, once an article is published in some approved venue, it should be taken as truth

Neuroskeptic, in 2008. Shift towards "internal criticism" of science.

https://retractionwatch.com/ in 2011. Anonymous peer review??? https://pubpeer.com/static/about

“the pervasiveness and persistence of the elderly stereotype” When the authors protest that none of the errors really matter, it makes you realize that, in these projects, the data hardly matter at all.

this smells like the authors defending this have this familiar pattern where they KNOW this thing is true, are just struggling to find the data to prove it, and feel like these critiques are just nitpicking. This is the steelman for a lot of effects people have firsthand experience of but we haven't scientifically proven/accepted. This came up a lot when I tweeted the "science confirms birds have a vocabulary! they are talking to each other!"

I am somewhat sympathic that we should take these things more seriously. I think it's OK to propose a wacky model as long as it gives you some predictive power that you can make use of. But I do think it is dishonest to justify it with bogus data? Reminds me of plurality of systems of knowledge in "Is Water H20"

(also Steve Levitt talks a bit about that, in how he feels the rigor of data was sometimes making him as a researcher lose the forest for the trees? in an interview with nate silver)

I take advantage of this sort of unsolicited free criticism to make my work better

🥇

[DefenderOfBasic]

The Nib – Repeat After Me (comic)

"the results are probably true, but human behavior is very varied so it's hard to get sufficient proof" -> interesting take about ego depletion. What should the average person do here, regarding this? I think it's wrong to take it as truth, and then take it as false, as you hear these headlines. I think it's ok to consider "is this model useful?"

don't forget that human behavior is itself affected by the belief about itself!

PLOS and open data!! https://plos.org/open-science/open-data/

[DefenderOfBasic]

Richard Feynman – Cargo Cult Science

reading this again now, I think I disagree with Feynman here. He criticizes this idea that it's dishonest when you ONLY publish what agrees with your theory:

In other words, publication probability depends upon the answer. That should not be done.

I think what he's asking for is in general a good thing but it may not be possible. Where can one find this mythical, unbiased researcher? Look, you said it's a bad thing that people doubted the erroneous numbers they got, even though they were correct:

When they got a number that was too high above Millikan’s, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan’s value they didn’t look so hard

but, sir! you have a selection bias here. When people looked at the result that defied expectation, and ignored it, they were WRONG in this case because that was the truth. But what about the MILLION ways things go wrong EVERY DAY that are NOT truth, that are the result of mistakes?

This happens in software too y'know. A clearly false thing can perpetuate, because it makes sense, it fits the theory. Not everyone has time to thoroughly check every single assumption. I think someone should. Teaching people this general process is good. Having a small % of feynman-like skeptics is good. AND the punch-line is, adversarial incentives. One scientist tries their hardest to prove the thing, another tries their hardest to tear it down. Science as lawyers, to surface the truth.

[DefenderOfBasic]

Andrew Gelman – Why is the scientific replication crisis centered on psychology?

If an economist makes a dramatic claim, journalists can call up experts on the left and the right and present a nuanced view

this actually re-inforces the thing I was saying. Andrew says that psychology has a bigger replication crises, potentially because economics research is more politically controversial, suggesting greater scrutinize (so it's that "lawyer model")

Speculation and data exploration are fine with me; indeed, they’re a necessary part of science. My problem with those papers is that they presented speculation as mature theory, that they presented data exploration as confirmatory evidence

honestly, this is a symptom of us missing alternative systems of knowledge. These people ARE truth seeking, it just doesn't fit within science and there's nowhere else to go that is rigorous.

[DefenderOfBasic]

⁉️ BIG question here is: what of effects that have ENORMOUS differences for specific cohorts? like growth mindset. If you apply this to 1000 people, and it has a life changing effect for 100, the overall result appears insignificant. Then the papers say "debunked" and you, one of the 100, are forced to dismiss something that YOU KNOW IS TRUE WITH FIRSTHAND KNOWLEDGE