Add examples for typical use cases

[paulsharpeY]

I installed Superpower as I'm not aware of any other R package which can do a power calculation for a 2x2 mixed ANOVA. My use case is to reproduce the following power calculation:

A priori power analysis revealed a sample size of 52 would adequately power ([1-β] = 0.80; ⍺ = 0.05, two-tailed) an effect size of f = 0.27 for a 2 (group) by 2 (time) mixed repeated- measures ANOVA interaction term (Faul et al. 2009). This effect size was based on a meta-analytic data for our variables of interest (i.e., cognition, attention, and self-reported mindfulness; Eberth and Sedlmeier 2012).

Having read the Superpower docs, I don't appear to have some of the numbers (means, SDs) required.

1. Given the information I have, can I calculate the n required to reliably measure this interaction? I want to check the n=52 claim. If so, how do I do that?
2. Could the docs be updated to reflect the answer to the previous question, as I suspect that this is a common use case.

[Lakens]

Hi,

The problem is you are trying to check an inadequate power analysis. That being said, I recognize this is a common use case, and we can think about how to accommodate it. The problem is the original authors performed a mindless power analysis – plugging in a number (f = 0.27) that does not mean much. One thing you can do is to create means and SD that would yield a f = 0.27 for a 2x2 mixed interaction. I think writing a convenience function for this would be useful. For now, you can enter means and sd, and then simply change 1 mean in the design until f = 0.27 (as indicated by ANOVA_exact.

Does that help?

Daniel

[paulsharpeY]

Maybe. Do I also need to invent a correlation value for the w/s factor? If so, what should I use? Can I try out this 'empirical replication approach' using the shiny app?

Here's my story, which I suspect is an instance of a common use case (hence the inadequate power calculation cited!):

• A priori power calculations give you confidence you're testing enough participants
• You do this by estimating an effect size relevant to your study, ideally from a meta-analysis
• You want to do this R rather than G*Power because everything
• One of the pwr functions will calculate n if you supply effect size, alpha and power
• pwr is for single factor designs (cf. this Cross Validated question)
• At this point you'll probably find Superpower via Google.

Some examples that talk to this (anti) pattern would help. Finding a reliable effect size always seems difficult. Perhaps using cell means, common SD and w/s correlations is easier, but an example pattern would be useful to indicate how you might approach finding reliable estimates for these numbers.

:)

[Lakens]

How did you get the f value? What was your starting point? Maybe we can explain how to ignore the f and define what you predict in means, sd and r. The philosophy of Superpower is that f values are typically not a good starting point. If you feel you need to 'invent' the r, you might not be ready to test a hypothesis.

---

From: paulsharpeY notifications@github.com Sent: Thursday, February 27, 2020 1:41:52 PM To: arcaldwell49/Superpower Superpower@noreply.github.com Cc: Lakens, D. D.Lakens@tue.nl; Comment comment@noreply.github.com Subject: Re: [arcaldwell49/Superpower] Add examples for typical use cases (#12)

Maybe. Do I also need to invent a correlation value for the w/s factor? If so, what should I use? Can I try out this 'empirical replication approach' using the shiny apphttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Farcstats.io%2Fshiny%2Fanova-power%2F&data=02%7C01%7Cd.lakens%40tue.nl%7Ce6fd92fd11e44ebc308e08d7bb826d0f%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637184041158264577&sdata=%2F6A%2BpGfdepYrJyatNc1t0TJGSYIYdJAZa3T%2FN45459Q%3D&reserved=0?

Here's my story, which I suspect is an instance of a common use case (hence the inadequate power calculation cited!):

• A priori power calculations give you confidence you're testing enough participants
• You do this by estimating an effect size relevant to your study, ideally from a meta-analysis
• You want to do this R rather than G*Power because everything
• One of the pwr functions will calculate n if you supply effect size, alpha and power
• pwr is for single factor designs (cf. this Cross Validated questionhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstats.stackexchange.com%2Fquestions%2F432367%2Fr-power-analysis-for-a-2-by-2-within-within-anova-interaction-effect-pwrpwr%2F451607%23451607&data=02%7C01%7Cd.lakens%40tue.nl%7Ce6fd92fd11e44ebc308e08d7bb826d0f%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637184041158264577&sdata=1MncIRfrC%2B4qE74khrW74L0U40Dydfj2qaSJcnGRKOY%3D&reserved=0)
• At this point you'll probably find Superpower via Google.

Some examples that talk to this (anti) pattern would help. Finding a reliable effect size always seems difficult. Perhaps using cell means, common SD and w/s correlations is easier, but an example pattern would be useful to indicate how you might approach finding reliable estimates for these numbers.

:)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Farcaldwell49%2FSuperpower%2Fissues%2F12%3Femail_source%3Dnotifications%26email_token%3DABRQPYIOUDOCDK6FUTP3SPLRE6YJBA5CNFSM4K4ZIL62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENEHHCQ%23issuecomment-591950730&data=02%7C01%7Cd.lakens%40tue.nl%7Ce6fd92fd11e44ebc308e08d7bb826d0f%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637184041158274534&sdata=MNiCz69tJJ1fSkwl5H2vSzPAZLVvzML7RMVasKbK7I0%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABRQPYJ73EL452DPX4SPVZLRE6YJBANCNFSM4K4ZIL6Q&data=02%7C01%7Cd.lakens%40tue.nl%7Ce6fd92fd11e44ebc308e08d7bb826d0f%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637184041158274534&sdata=7Q4Ad2XY9fgFQ6fWNqGYSfRR%2Bbw85JfQO4lomaqLK0s%3D&reserved=0.

[paulsharpeY]

The f came from a paper I was reading (see direct quote above). I thought "I don't trust that power calculation, let's try to replicate it.". All I had to go on was what was in the paper :-/.

[arcaldwell49]

Just to make sure @paulsharpeY, you have seen our chapter on mixed ANOVAs correct?

I would agree with Daniel, that you can setup a design with means, SD and the within-subject correlation based on the underlying measurement. My suggestion would be this: create at least three different scenarios where the correlation is small, moderate, or large, and then find the mean difference required to produce that effect. Then you can gauge how reasonable the original effect size is based on the required difference/variance/correlation that would be needed to produce such an effect.

But let me see if I can help a little bit.

In a quick examination of the meta-analysis, it appears the "effect" is around 0.5 SD (Cohen's d = .5). From that I can setup a 2b*2w design in Superpower.

sample_design = ANOVA_design("2b*2w", n=52, r=.6, mu = c(1,1.5,1,1), sd = 1)

When this design is passed onto ANOVA_exact it provides the interaction with a power of ~80%.

This can be accomplished in our ANOVA_exact Shiny app as well (link here).

However you will notice the SD and means are for convenience and do not reflect "real" values that would be collected in a study.

[paulsharpeY]

Hi Aaron/Daniel,

Thanks, this is helping. I have a few more questions. For context, is the d you refer to from Eberth and Sedlmeier (2012), i.e.

What have we learned about the effects of mindfulness meditation by now? In our meta-analysis, we found a weighted mean correlation of r = 0.27, which corresponds to d = 0.56 (for comparison with earlier summaries that mainly used d, we occasionally report both effect sizes).

• Is their (and presumably Cohen's) f, the same as "a weighted mean correlation"?
• I assume this is cohen_f in the results from ANOVA_exact?
• Did d ~= 0.5 influence your choice of argument values in your ANOVA_design(...) above?
• For mixed designs, where the ANOVA_design man page says n Sample size in each condition, does this mean 'Sample size in each within subjects condition'? (Similarly with the shiny app where there's a single "Sample Size per Cell" field). I notice that this is how that argument is worded in the book.

[arcaldwell49]

Again, these are relatively made up numbers. I do not know what values or underlying assumptions the paper you quoted in the first post made about their power analysis. I demonstrated what it could look like if implemented in Superpower.

• Is their (and presumably Cohen's) f, the same as "a weighted mean correlation"? -- No.

• I assume this is cohen_f in the results from ANOVA_exact? -- Yes, if you reproduce the design I displayed in the screenshot it will reproduce this effect size

• Did d ~= 0.5 influence your choice of argument values in your ANOVA_design(...) above? -- yes. Any change in the sample size, mean difference, variance, or covariance (correlation) will affect power. Therefore, given a 0.5 SD difference at "post" between groups (.5 increase in mean in one group with a common SD of 1), and given a sample size of 52, the remaining modifiable factor is the correlation (since this is design involves a within-subjects factor). I have no idea whether any of these assumptions are reasonable in this area of research.

• For mixed designs, where the ANOVA_design man page says n Sample size in each condition, does this mean 'Sample size in each within subjects condition'? (Similarly with the shiny app where there's a single "Sample Size per Cell" field). I notice that this is how that argument is worded in the book. -- The n is the number of observations within each cell of the design. For a 2b*2w design with a n = 52 this would mean there are 52 subjects in each between subjects-group (total of 104) and the number of observations within each within-subject condition (pre-to-post). If it was a all within subjects design (say 2w*2w) then 52 would be the total number of participants.

Also, if you are planning a 2b*2w randomized design I would suggest considering an ANCOVA (baseline as a co-variate) for the final analysis https://doi.org/10.1016/j.jclinepi.2006.02.007