pdeninis
commented
2 years ago Maybe I have a reasonable answer.
I tried to design both a single- and a multi-stage design with equal characteristic (except than the multiplicity correction, that is not [yet?] provided for multi-stage design), i.e.: Bernoulli, power 80%, alpha 0.05%, control rate=0.33, d1=0.33, d0=0, type of control=MARGINAL.
In the single-stage, the output is below:
Design summary
Inputs
The following choices were made:
K=3 experimental treatments will be included in the trial.
A significance level of α=0.05 will be used, in combination with the Holm-Bonferroni correction.
The response rate in the control arm will be assumed to be: π0=0.33 .
The marginal power for each null hypothesis will be controlled to level 1−β=0.8 under each of their respective least favourable configurations.
The interesting and uninteresting treatment effects will be: δ1=0.33 and δ0=0 respectively.
The target allocation to each of the experimental arms will be: the same as the control arm.
The sample size in each arm will be required to be an integer.
Plots will be produced.
Outputs
The total required sample size is: N=148 .
The required sample size in each arm is: (n0,…,nK)=(37,37,37,37) .
Therefore, the realised allocation ratios to the experimental arms are: (r1,…,rK)=(1,1,1) .
The maximum familywise error-rate is: 0.043 .
The minimum marginal power is: 0.81 .
The following critical thresholds should be used with the chosen multiple comparison correction: (0.017,0.025,0.05) . In the multi-stage, instead it is:
Design summary
Inputs
The following choices were made:
K=3 experimental treatments will initially be included in the trial.
A maximum of J=2 stages will be allowed in the trial.
A separate stopping rule will be used.
The stage-wise sample size will be variable.
The lower stopping boundaries will be Pocock.
The upper stopping boundaries will be Pocock.
A family-wise error-rate of α=0.05 will be used.
The minimum marginal power will be controlled to level 1−β=0.8 under each of their respective least favourable configurations.
The response rate in the control arm will be assumed to be: π0= .
The interesting and uninteresting treatment effects will be: δ1=0.33 and δ0=0 respectively.
The target allocation to each of the experimental arms will be: the same as the control arm.
The sample size in each arm will be required to be an integer.
Plots will be produced.
Outputs
The maximum required sample size is: 160 .
The stage-wise sample size to the control arm is: 20.
The realised allocation ratios to the experimental arms are: (r1,…,rK)=(1,1,1) .
The maximum familywise error-rate is: 0.05 .
The minimum marginal power is: 0.818 .
The lower stopping boundaries are: (−2.28,2.28) .
The upper stopping boundaries are: (2.28,2.28) .My conclusion is that the sequential familywise error rate for the multi-stage design should be exactly 0.05, just as the Output says.
I was in doubt since I was not being able to individuate the penalization for the sequential design. By comparing the two, though, I noted that the required sample-size for single-stage was 37 per group, total=148, while for multi-stage it is 20+20=40, total = 160. Maybe the 3 units per group are the cost of two-stage design? If it were so, it would be a very interesting price!
The output seems now reasonably clear: as I expected, in the multi-stage case the app provides only the calculation for the sequentiality correction.
Am I wrong?
PS As you can see, the row:
The response rate in the control arm will be assumed to be: π0= .
does not show the entered value (0.33).
Moreover, in downloading the pdf and docx reports, the tables are not those reported by the web app:
π0 π1 π2 π3 Pdis Pcon P1 P2 P3 FWER PHER ESS
HG 0.33 0.33 0.33 0.33 0.05 0.001 0.019 0.019 0.019 0.05 0.019 158.626
HA 0.33 0.66 0.66 0.66 0.959 0.638 0.818 0.818 0.818 0 0 127.156
LFC1 0.33 0.66 0.33 0.33 0.819 0.003 0.818 0.019 0.019 0.036 0.013 149.616
LFC2 0.33 0.33 0.66 0.33 0.819 0.003 0.019 0.818 0.019 0.036 0.013 149.617
LFC3 0.33 0.33 0.33 0.66 0.819 0.003 0.019 0.019 0.818 0.036 0.013 149.617
Equal #1 0.33 0 0 0 0 0 0 0 0 0 95.633
Equal #2 0.33 0.007 0.007 0.007 0 0 0 0 0 0 99.052
Equal #3 0.33 0.013 0.013 0.013 0 0 0 0 0 0 102.136
Equal #4 0.33 0.02 0.02 0.02 0 0 0 0 0 0 105.22
Equal #5 0.33 0.027 0.027 0.027 0 0 0 0 0 0 108.305but look like these:
pi0 pi1 pi2 pi3 FWERI1 FWERI2 FWERI3 FWERII1 FWERII2 FWERII3
0 0 0 0 0 0 0 0 0 0
0 1 1 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0 0
0 0 0 1 0 0 0 0 0 0
pi0 pi1 pi2 pi3 Pdis Pcon P1 P2 P3
0 0 0 0 0 0 0 0 0
0 1 1 1 1 1 1 1 1
0 1 0 0 1 0 1 0 0
0 0 1 0 1 0 0 1 0
0 0 0 1 1 0 0 0 1
pi0 pi1 pi2 pi3 PHER FDR pFDR FNDR Sens Spec ESS SDSS MeSS MoSS maxN
0 0 0 0 0 0 1 0 0 1 159 6 160 160 160
0 1 1 1 0 0 0 0 1 0 127 29 140 160 160
0 1 0 0 0 0 0 0 1 1 150 11 160 160 160
0 0 1 0 0 0 0 0 1 1 150 11 160 160 160
0 0 0 1 0 0 0 0 1 1 150 11 160 160 160
mjg211
Dear professor Grayling,
As I anticipated in the previous post, I am a little embarrassed in interpreting the tables printed. I know it is surely my fault, not having studied in depth your project, that is complex indeed.
Until I will be more aware of its thorough working, to provisionally simplify (I hope not unduly!), my aim is to get a kind of local alpha taking account for sequentiality to guarantee "the overall test" to stay below the, say, alpha=0.05. I would enter it as a sequentially-adjusted alpha level into any kind of multiple comparison I will do next.
As an example I report the output of the rpact application, which though does not allow for multi-arm design.
Here, in my understanding, the wanted alpha level for the first interim would be 0.0294, that should be corrected for sequentiality only, being it relative to a two-arm design. It is equal to the final stage, having been chosen the Pocock's design. Setting the O'Brien & Fleming design the two numbers become:
Two-sided local significance level 0.0052 0.0480in agreement with my expectations.
In the "Operating characteristics summary: "(key, error rates and others) I am unsure to correctly interpret the big deal of data so, to be safe, I would prefer to use a more simple and well-identified datum.
Please, may you tell me what is the number I am looking for? In alternative, may you tell me which numbers should I look at to get the alpha level to use with the omnibus test and with the direct comparisons between control and experimental groups?
Thank you in advance! Paolo
PS This information is easily found even in your function in case of one-stage design, in which the multiplicity correction method can be chosen. As an example:
Here the output is clearer: sample size in each arm = 18, critical thresholds for multiple comparisons with the step-down method = (0.017,0.025,0.05) in particular. I am not sure, however, what exactly the "maximum familywise error rate" = 0.043 represents, i.e. why it is not 0.05 (I believe it is something may be overlooked for it is only linked to the three comparisons with the Bonferroni-Holm's method, since it seems independent from specific data).
In the sequential section I would have expected something similar. However, the correction method is no longer selectable. I guess there is a reason for this, but I hope there is a way to get the equivalent information, maybe in the tables ...
As an instance, if the single-stage design above were to become a stage of two (for a total N=72*2=144 samples) having the aim of decreasing the effect to detect but allowing to stop earlier the trial in case a more substantial one was present, by how much the multiple comparisons thresholds of the Bonferroni-Holm's correction would be penalized in the two stages using the Pocock design? Is there a simple way to answer this question with your package?