Open ddscott2 opened 3 years ago
100% power is fairly common when you have a large amount of data relative to the effect size.
0% power is usually due to errors during simulations, so looking at the error logs might help.
There seems to be something weird happening with ID, looking at the "0 rows" in the first image and the NA's in the second.
Here's my output after running the "$warnings" and "$errors" commands...
> powercurve
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 100.0% (99.63, 100.0) - 10200 rows
75: 100.0% (99.63, 100.0) - 15200 rows
NA: 34.30% (31.36, 37.34) - 17600 rows
NA: 0.00% ( 0.00, 0.37) - 17600 rows
Time elapsed: 27 h 39 m 51 s
> powercurve$warnings
> powercurve$errors
Here's what I ran:
powersim <- powerSim(SkillTwoWay2, fixed("Object:OP.c"), nsim = 10)
powercurve <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50, 75, 100, 125))
Output for powersim:
> powersim
Power for predictor 'Object:OP.c', (95% confidence interval):
100.0% (69.15, 100.0)
Test: z-test
Effect size for Object:OP.c is 0.33
Based on 10 simulations, (1 warning, 0 errors)
alpha = 0.05, nrow = 17600
Time elapsed: 0 h 12 m 12 s
nb: result might be an observed power calculation
Look like you're running out of memory. If powerSim
is coping you might be better off running a separate powerSim
for each sample size?
I ran the first two sample sizes, with the $warnings and $error reports too:
> powercurve1 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50))
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation===================================================================================================================================================================================|
> powercurve2 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(75))
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation===================================================================================================================================================================================|
> powercurve1
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 0.00% ( 0.00, 0.37) - 0 rows
Time elapsed: 0 h 26 m 41 s
> powercurve2
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
75: 0.00% ( 0.00, 0.37) - 0 rows
Time elapsed: 0 h 26 m 45 s
> powercurve1$warnings
[1] stage index message
<0 rows> (or 0-length row.names)
> powercurve1$errors
> powercurve2$warnings
[1] stage index message
<0 rows> (or 0-length row.names)
> powercurve2$errors
Did I correctly run a powerSim for each sample size?
One of your problems is that you appear to have only 92 ID's in your data set and you're trying to set breaks at 100 and 125.
I don't know why you're getting those "0 rows" results though. Could you please try the following:
str(naomit$ID)
sort(unique(naomit$ID))
I'd also recommend setting nsim
when you're trying out powerCurve
(like you have with powerSim
) and not run large values of nsim
until you have it working.
> str(naomit$ID)
Factor w/ 92 levels "0qajoaqo","100",..: 1 1 1 1 1 1 1 1 1 1 ...
> sort(unique(naomit$ID))
[1] 0qajoaqo 100 101 102 103 104 106 107 108 109 111 112 113 114 115
[16] 116 117 118 119 120 121 122 123 124 1ia2528x 1nshvpsy 1rr3xfrc 1w9jcj6d 2vlt2v2n 469ikial
[31] 5cl2qh5d 6mprqjfs 72jrug5w 79r9ob8q 82f94y0r 8g8c8b0c 8o6zgmlr 8s46ki0d bpii6i6t c1ihbqon d07miqx5 d3u0mv82 d4pzhzuv doujtcdp eqqowmki
[46] fd0ppmhk fhe60ywj fiw7o4le g4grw9v4 gu4pmqdo hqs5z3ny iee5otw5 irnf7ptw itembr31 iu23r0b8 iwg56hov j8arfng0 kz43a4li lmkj9lxu mgyg0zch
[61] ml05q0f1 mviyrtuh mzo5vci3 ne7fqm9r nubr18c2 o82g6766 ojnj4qff onxsdarj pt93mnno qhcpszrl qy6ihy95 rtlxrm22 rzigt3jr saq3btyr sblyczt0
[76] t2lpq7ym u3z3slfe vvwjon74 w15c8joa wpux51u5 yn36nefa yvmh7j75 ziedvces ziql5otu zmbmxdsx zqqewrvy zyux38sq
92 Levels: 0qajoaqo 100 101 102 103 104 106 107 108 109 111 112 113 114 115 116 117 118 119 120 121 122 123 124 1ia2528x 1nshvpsy ... zyux38sq
> powercurve1 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50), nsim = 10)
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation=======================================================================================|
> powercurve2 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(75), nsim = 10)
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation=======================================================================================|
> powercurve1
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 100.0% (69.15, 100.0) - 10200 rows
Time elapsed: 0 h 4 m 31 s
> powercurve2
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
75: 100.0% (69.15, 100.0) - 15200 rows
Time elapsed: 0 h 8 m 28 s
Here's the output after I included nsim
Now this is very strange. Has anything changed between running these two?
> powercurve1 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50))
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation===================================================================================================================================================================================|
> powercurve1
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 0.00% ( 0.00, 0.37) - 0 rows
Time elapsed: 0 h 26 m 41 s
> powercurve1 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50), nsim = 10)
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation=======================================================================================|
> powercurve1
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 100.0% (69.15, 100.0) - 10200 rows
Time elapsed: 0 h 4 m 31 s
Is this reproducible? If you run them one immediately after the other, with only nsim changed, does the same thing happen?
table(naomit$ID)
might be useful later too please.
I ran them immediately after the other, this is what I got:
> powercurve <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50))
Calculating power at 1 sample sizes along ID
(1/1) Simulating: |========== (1/1) Simulating: |============ (1/1) Simulating: |==================== (1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |===============================================================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================(1/1) Simulating: |======================This appears to be an "observed power" calculation=======================================================================================|
> powercurve1 <- powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50), nsim = 10)
Calculating power at 1 sample sizes along ID
This appears to be an "observed power" calculation=======================================================================================|
> powercurve
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 100.0% (99.63, 100.0) - 10200 rows
Time elapsed: 8 h 34 m 38 s
> powercurve1
Power for predictor 'Object', (95% confidence interval),
by number of levels in ID:
50: 100.0% (69.15, 100.0) - 10200 rows
Time elapsed: 0 h 5 m 13 s
Here's what I get after running table(naomit$ID)
> table(naomit$ID)
0qajoaqo 100 101 102 103 104 106 107 108 109 111 112 113 114 115 116 117 118
200 200 200 200 200 200 200 200 200 200 200 200 200 200 400 200 200 200
119 120 121 122 123 124 1ia2528x 1nshvpsy 1rr3xfrc 1w9jcj6d 2vlt2v2n 469ikial 5cl2qh5d 6mprqjfs 72jrug5w 79r9ob8q 7rbd77vi 82f94y0r
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 0 200
8embhxsq 8g8c8b0c 8o6zgmlr 8s46ki0d bpii6i6t bpx1s04c c1ihbqon d07miqx5 d3u0mv82 d4pzhzuv doujtcdp eqqowmki fd0ppmhk fhe60ywj fiw7o4le g4grw9v4 gu4pmqdo hqs5z3ny
0 200 200 200 200 0 200 200 200 200 200 200 200 200 200 200 200 200
iee5otw5 irnf7ptw itembr31 iu23r0b8 iwg56hov j8arfng0 kz43a4li lmkj9lxu mgyg0zch ml05q0f1 mviyrtuh mzo5vci3 ne7fqm9r nubr18c2 o82g6766 ojnj4qff onxsdarj pt93mnno
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
qhcpszrl qy6ihy95 rb1phxuq rtlxrm22 rzigt3jr saq3btyr sblyczt0 t2lpq7ym u3z3slfe vvwjon74 w15c8joa wpux51u5 yd0oewno yn36nefa yvmh7j75 ziedvces ziql5otu zmbmxdsx
200 200 0 200 200 200 200 200 200 200 200 200 0 200 200 200 200 200
zqqewrvy zyux38sq
200 200
Looks like that's working now? Can you remember what you changed?
If you want to test for 100 or 125 participants you will probably need to use extend
. You should be careful with your unbalanced data though, I would recommend removing ID 115 and running droplevels
on your ID factor.
Hi, I have the same / very similar issue. I have a dataset were 150 participants rated 80 stimuli (Trials), which all differed on three variables (populistic: 0 or 1; false : 0 or 1; and cognitiveutility: 0 or 1). A trial can have all possible combinations on these three variables and data is balanced.
I have a pilot data set of about 150 participants that was sufficient observe effects of each factor:
model1 <- lmer(susceptibility ~ False + Populistic + CognitiveUtility + (1|Participant) + (1|Trial), datafile)
summary(model1)
linear mixed model fit by REML. t-tests use Satterthwaite's method ['lmerModLmerTest'] Formula: susceptibility ~ False + Populistic + CognitiveUtility + (1 | Participant) + (1 | Trial) Data: datafile
REML criterion at convergence: 38771.8
Scaled residuals: Min 1Q Median 3Q Max -3.5929 -0.5980 -0.0350 0.5481 3.7207
Random effects: Groups Name Variance Std.Dev. Participant (Intercept) 1.13120 1.0636
Trial (Intercept) 0.09113 0.3019
Residual 1.22322 1.1060
Number of obs: 12466, groups: Participant, 156; Trial, 80Fixed effects: Estimate Std. Error df t value Pr(>|t|)
(Intercept) 2.82633 0.11046 221.95525 25.587 < 2e-16 False1 -0.26731 0.07035 76.00739 -3.800 0.00029 Populistic1 -0.31432 0.07035 76.00739 -4.468 2.71e-05 CognitiveUtility1 0.59458 0.07035 76.00739 8.452 1.50e-12Signif. codes: 0 ‘’ 0.001 ‘’ 0.01 ‘’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects: (Intr) False1 Pplst1 False1 -0.318
Populistic1 -0.319 0.000
CgntvUtlty1 -0.319 0.000 0.000
So far so good. What I want to do next, is to test whether I can reduce the number of Trials, while still have sufficient power to observe one of the effects (False).
So I run the following code (low nsim as I am testing things out):
p_curve_model <- powerCurve(model1, test = fixed("False"), along="Trial", breaks=c(10,20,30,40, 50,60,70,80), nsim = 20)
The problem is that I always get the following results where Power suddently goes from 0 to more when I reach 50 trials:
Power for predictor 'False', (95% confidence interval), by number of levels in Trial: 10: 0.00% ( 0.00, 16.84) - 1560 rows 20: 0.00% ( 0.00, 16.84) - 3120 rows 30: 0.00% ( 0.00, 16.84) - 4680 rows 40: 0.00% ( 0.00, 16.84) - 6240 rows 50: 60.00% (36.05, 80.88) - 7800 rows 60: 100.0% (83.16, 100.0) - 9360 rows 70: 100.0% (83.16, 100.0) - 10920 rows 80: 100.0% (83.16, 100.0) - 12480 rows
Time elapsed: 0 h 0 m 59 s
I also get the following error messages:
stage index message
1 Fitting 1 contrasts can be applied only to factors with 2 or more levels 2 Fitting 2 contrasts can be applied only to factors with 2 or more levels 3 Fitting 3 contrasts can be applied only to factors with 2 or more levels 4 Fitting 4 contrasts can be applied only to factors with 2 or more levels 5 Fitting 5 contrasts can be applied only to factors with 2 or more levels 6 Fitting 6 contrasts can be applied only to factors with 2 or more levels 7 Fitting 7 contrasts can be applied only to factors with 2 or more levels 8 Fitting 8 contrasts can be applied only to factors with 2 or more levels 9 Fitting 9 contrasts can be applied only to factors with 2 or more levels 10 Fitting 10 contrasts can be applied only to factors with 2 or more levels 11 Fitting 11 contrasts can be applied only to factors with 2 or more levels 12 Fitting 12 contrasts can be applied only to factors with 2 or more levels 13 Fitting 13 contrasts can be applied only to factors with 2 or more levels 14 Fitting 14 contrasts can be applied only to factors with 2 or more levels 15 Fitting 15 contrasts can be applied only to factors with 2 or more levels 16 Fitting 16 contrasts can be applied only to factors with 2 or more levels 17 Fitting 17 contrasts can be applied only to factors with 2 or more levels 18 Fitting 18 contrasts can be applied only to factors with 2 or more levels 19 Fitting 19 contrasts can be applied only to factors with 2 or more levels 20 Fitting 20 contrasts can be applied only to factors with 2 or more levels 21 Fitting 1 contrasts can be applied only to factors with 2 or more levels 22 Fitting 2 contrasts can be applied only to factors with 2 or more levels 23 Fitting 3 contrasts can be applied only to factors with 2 or more levels 24 Fitting 4 contrasts can be applied only to factors with 2 or more levels 25 Fitting 5 contrasts can be applied only to factors with 2 or more levels 26 Fitting 6 contrasts can be applied only to factors with 2 or more levels 27 Fitting 7 contrasts can be applied only to factors with 2 or more levels 28 Fitting 8 contrasts can be applied only to factors with 2 or more levels 29 Fitting 9 contrasts can be applied only to factors with 2 or more levels 30 Fitting 10 contrasts can be applied only to factors with 2 or more levels 31 Fitting 11 contrasts can be applied only to factors with 2 or more levels 32 Fitting 12 contrasts can be applied only to factors with 2 or more levels 33 Fitting 13 contrasts can be applied only to factors with 2 or more levels 34 Fitting 14 contrasts can be applied only to factors with 2 or more levels 35 Fitting 15 contrasts can be applied only to factors with 2 or more levels 36 Fitting 16 contrasts can be applied only to factors with 2 or more levels 37 Fitting 17 contrasts can be applied only to factors with 2 or more levels 38 Fitting 18 contrasts can be applied only to factors with 2 or more levels 39 Fitting 19 contrasts can be applied only to factors with 2 or more levels 40 Fitting 20 contrasts can be applied only to factors with 2 or more levels 41 Fitting 1 contrasts can be applied only to factors with 2 or more levels 42 Fitting 2 contrasts can be applied only to factors with 2 or more levels 43 Fitting 3 contrasts can be applied only to factors with 2 or more levels 44 Fitting 4 contrasts can be applied only to factors with 2 or more levels 45 Fitting 5 contrasts can be applied only to factors with 2 or more levels 46 Fitting 6 contrasts can be applied only to factors with 2 or more levels 47 Fitting 7 contrasts can be applied only to factors with 2 or more levels 48 Fitting 8 contrasts can be applied only to factors with 2 or more levels 49 Fitting 9 contrasts can be applied only to factors with 2 or more levels 50 Fitting 10 contrasts can be applied only to factors with 2 or more levels 51 Fitting 11 contrasts can be applied only to factors with 2 or more levels 52 Fitting 12 contrasts can be applied only to factors with 2 or more levels 53 Fitting 13 contrasts can be applied only to factors with 2 or more levels 54 Fitting 14 contrasts can be applied only to factors with 2 or more levels 55 Fitting 15 contrasts can be applied only to factors with 2 or more levels 56 Fitting 16 contrasts can be applied only to factors with 2 or more levels 57 Fitting 17 contrasts can be applied only to factors with 2 or more levels 58 Fitting 18 contrasts can be applied only to factors with 2 or more levels 59 Fitting 19 contrasts can be applied only to factors with 2 or more levels 60 Fitting 20 contrasts can be applied only to factors with 2 or more levels 61 Fitting 1 contrasts can be applied only to factors with 2 or more levels 62 Fitting 2 contrasts can be applied only to factors with 2 or more levels 63 Fitting 3 contrasts can be applied only to factors with 2 or more levels 64 Fitting 4 contrasts can be applied only to factors with 2 or more levels 65 Fitting 5 contrasts can be applied only to factors with 2 or more levels 66 Fitting 6 contrasts can be applied only to factors with 2 or more levels 67 Fitting 7 contrasts can be applied only to factors with 2 or more levels 68 Fitting 8 contrasts can be applied only to factors with 2 or more levels 69 Fitting 9 contrasts can be applied only to factors with 2 or more levels 70 Fitting 10 contrasts can be applied only to factors with 2 or more levels 71 Fitting 11 contrasts can be applied only to factors with 2 or more levels 72 Fitting 12 contrasts can be applied only to factors with 2 or more levels 73 Fitting 13 contrasts can be applied only to factors with 2 or more levels 74 Fitting 14 contrasts can be applied only to factors with 2 or more levels 75 Fitting 15 contrasts can be applied only to factors with 2 or more levels 76 Fitting 16 contrasts can be applied only to factors with 2 or more levels 77 Fitting 17 contrasts can be applied only to factors with 2 or more levels 78 Fitting 18 contrasts can be applied only to factors with 2 or more levels 79 Fitting 19 contrasts can be applied only to factors with 2 or more levels 80 Fitting 20 contrasts can be applied only to factors with 2 or more levels
I don't understand this, because there are no variables with less than 2 levels.
str(datafile$Trial) Factor w/ 80 levels "1","2","3","4",..: 1 2 3 4 5 6 7 8 9 10 ... str(datafile$Participant) Factor w/ 156 levels "1","3","4","6",..: 1 1 1 1 1 1 1 1 1 1 ... str(datafile$False) Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ... str(datafile$Populistic) Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ... str(datafile$CognitiveUtility) Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
Any help would be highly appreciated!
I found the problem. Given the data structure, until Trial 40, one variable is constant (i.e., 1 level). I was assuming that each simulation draws a random sample from Trial, but that seems not to be the case.
If Trial level is = 40 for instance, it will only draw random samples (I assume with replacement) from the first 40 trials. I would wish a function that rather selects randomly 40 trials from the total 80 trials available.
Hello,
My colleague and I have been running different variations of a power analysis and run into an issue where the power is either 100% or 0%.
Here is our model:
Model specification:
Model info:
Some attempted models and output
powerSim(SkillTwoWay2, fixed("Object:OP.c"), nsim = 10)
- Issue: observed power is 100%powerSim(SkillTwoWay, fixed("Object:OP.c"), nsim = 10)
- Issue: observed power is 0%powerCurve(SkillTwoWay2, fixed("Object"), along = "ID", breaks=c(50, 75, 100, 125))
- Issue: the break at 50 provides a power of 78.3%, but 0% at the 75 break, and NA for the breaks for 100 and 125powerCurve(SkillTwoWay, fixed("Object"), along = "ID")