search

ecsalomon / TSR---Test-Stats-Replication

Test Statistics Based Research Quality Metrics and Replication in Psychology
2 stars 0 forks source link

Coding surprise #3

Open soodoku opened 8 years ago

soodoku commented 8 years ago

Dear E,

I think the information that I would look in the paper would be the 'surprise' --- how unlikely to detect an effect of size X given power.

And then plot by replication?

ecsalomon commented 8 years ago

Do you mean the post-hoc power of the original effect? I think that this might be captured by the r-index, which is:

Post-hoc power - (Proportion of results that are significant - Post hoc power)

The (Proportion of results that are significant - Post hoc power) is the "inflation": how many more/fewer results were observed than would be expected based on the power for the observed effect size. The r-index then takes the difference between the observed power and the inflation.

Inflation and r-index in this data set ate correlated at -.94, so I don't expect inflation to perform much differently from r-index.

soodoku commented 8 years ago

Right. I had noticed that but perhaps not digested.

The following may reflect my ignorance about the data at hand:

Perhaps my point is that counting up number of sig. results may be somewhat crude, as it loses a bunch of information. Combining information about effect size (per result), power for observing an effect of that size with the sample etc. may be a way forward.

ecsalomon commented 8 years ago

Hmmm. I see. What about something like the number of additional participants needed to achieve 80% power in a replication for the observed effect size in the original?

i.e., N for 80% power given observed effect size - actual N