search

pitakakariki / simr

Power Analysis of Generalised Linear Mixed Models by Simulation
68 stars 19 forks source link

Power is always 100% #275

Open hyc970828 opened 4 months ago

hyc970828 commented 4 months ago

Hi all, I'm new to GitHub so apologies if my entry is not formatted correcting for you to help me appropriately. Happy to edit/add info if needed! My experiment was a 2×3 mixed experiment, with 2（LEIBIE） being the intra-subject variable and 3（CONDITION） being the inter-subject variable. The number of subjects is 120 in total, 40 in each group. Each subject was required to complete 72 items, for a total of 8,480 items. My model is:

modelzero.corr <- glmer(CORR ~ LEIBIE*CONDITION+ (1|item)+ (1|ASUBJ),  data = data1, family="binomial")

and I tried:

PowerAB_ttest = simr::powerSim(fit = modelopt1.RT, test = fixed('CONDITION1:LEIBIE1', method = 'z'), nsim=5)

This runs and gives me this output:

> print(Powertask_Ftest)
Power for predictor 'LEIBIE1:CONDITION1', (95% confidence interval):
      100.0% (47.82, 100.0)
Test: z-test
      Effect size for LEIBIE1:CONDITION1 is -0.42
Based on 5 simulations, (0 warnings, 0 errors)
alpha = 0.05, nrow = 8480
Time elapsed: 0 h 0 m 28 s
nb: result might be an observed power calculation

I'm thinking maybe it's because there are too many projects? Do I specify the number of subjects? But I didn't know what to do.

pitakakariki commented 4 months ago

There are a large number of subjects and items so high power is not unexpected here. Given the large sample size, I'm not surprised to see 100% power given the effect size.

If the confidence interval around LEIBIE1:CONDITION1 excludes zero by a large margin, then you already know there is enough power to reject a null effect, and a power analysis gives you very little additional information.