search

Lakens / TOSTER

TOST equivalence test and power functions
35 stars 18 forks source link

TOSTtwo.prop question #48

Closed HeatherUrry closed 4 years ago

HeatherUrry commented 4 years ago

Hi Daniël,

As part of my attempt to get smarter about inferiority tests, I'm trying to replicate Example 4 from Lakens, Scheel, & Isager (2018), https://journals.sagepub.com/doi/pdf/10.1177/2515245918770963 using TOSTER.

The text says, "The inferiority test on the data from Lynott et al. reveals that one can reject effects larger than Δ = 0.25, z = –9.12, p < .001 (see Fig. 2d)" (p. 266).

But when I run the TOSTtwo.prop command (below), it gives a result of Z = -9.094, p < .001. I thought it might be rounding error, but that doesn't seem to be the case. I downloaded Lynott et al.'s data and calculated the proportions to 7 decimal places. What am I doing wrong? Alternatively, does this perhaps reflect a version issue? I'm using TOSTER v 0.3.4.

Thanks! (And thanks too for writing this package. It's really useful.)

Heather

TOSTtwo.prop(prop1 = 0.5074257, prop2 = 0.5745721, n1 = 404, n2 = 409 , low_eqbound = -1, high_eqbound = .25, alpha =.05)

TOST results: Z-value lower bound: 26.75      p-value lower bound: 0.000 Z-value upper bound: -9.09      p-value upper bound: 0.00000000000000000005   Equivalence bounds: low eqbound: -1 high eqbound: 0.25   TOST confidence interval: lower bound 90% CI: -0.125 upper bound 90% CI:  -0.01   NHST confidence interval: lower bound 95% CI: -0.136 upper bound 95% CI:  0.001   Equivalence Test based on Fisher's exact z-test Result: The equivalence test was significant, Z = -9.094, p = 0.0000000000000000000479, given equivalence bounds of -1.000 and 0.250 and an alpha of 0.05.   Null-Hypothesis Fisher's exact z-test Result: The null hypothesis test was non-significant, Z = -1.925, p = 0.0542, given an alpha of 0.05.   Based on the equivalence test and the null-hypothesis test combined, we can conclude that the observed effect is statistically not different from zero and statistically equivalent to zero.

Lakens commented 4 years ago

Hi, the manuscript you took the example from has fully reproducible code in R markdown. Can you please take a look at the code linked to in the disclosures? What happens if you take the test from our manuscript and compare it to the code you created?

From: HeatherUrry notifications@github.com Sent: Sunday, January 5, 2020 8:18:31 PM To: Lakens/TOSTER TOSTER@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [Lakens/TOSTER] TOSTtwo.prop question (#48)

Hi Daniël,

As part of my attempt to get smarter about inferiority tests, I'm trying to replicate Example 4 from Lakens, Scheel, & Isager (2018), https://journals.sagepub.com/doi/pdf/10.1177/2515245918770963https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjournals.sagepub.com%2Fdoi%2Fpdf%2F10.1177%2F2515245918770963&data=02%7C01%7CD.Lakens%40tue.nl%7Cf09fe977d0874746bc8608d792140f02%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637138487176929503&sdata=KYJb50MiBAvklyrF2yJwxChJ0YFCctZJka92zkCEMVc%3D&reserved=0 using TOSTER.

The text says, "The inferiority test on the data from Lynott et al. reveals that one can reject effects larger than Δ = 0.25, z = –9.12, p < .001 (see Fig. 2d)" (p. 266).

But when I run the TOSTtwo.prop command (below), it gives a result of Z = -9.094, p < .001. I thought it might be rounding error, but that doesn't seem to be the case. I downloaded Lynott et al.'s data and calculated the proportions to 7 decimal places. What am I doing wrong? Alternatively, does this perhaps reflect a version issue? I'm using TOSTER v 0.3.4.

Thanks! (And thanks too for writing this package. It's really useful.)

Heather

TOSTtwo.prop(prop1 = 0.5074257, prop2 = 0.5745721, n1 = 404, n2 = 409 , low_eqbound = -1, high_eqbound = .25, alpha =.05)

TOST results: Z-value lower bound: 26.75 p-value lower bound: 0.000 Z-value upper bound: -9.09 p-value upper bound: 0.00000000000000000005

Equivalence bounds: low eqbound: -1 high eqbound: 0.25

TOST confidence interval: lower bound 90% CI: -0.125 upper bound 90% CI: -0.01

NHST confidence interval: lower bound 95% CI: -0.136 upper bound 95% CI: 0.001

Equivalence Test based on Fisher's exact z-test Result: The equivalence test was significant, Z = -9.094, p = 0.0000000000000000000479, given equivalence bounds of -1.000 and 0.250 and an alpha of 0.05.

Null-Hypothesis Fisher's exact z-test Result: The null hypothesis test was non-significant, Z = -1.925, p = 0.0542, given an alpha of 0.05.

Based on the equivalence test and the null-hypothesis test combined, we can conclude that the observed effect is statistically not different from zero and statistically equivalent to zero.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FLakens%2FTOSTER%2Fissues%2F48%3Femail_source%3Dnotifications%26email_token%3DABRQPYIBJQ3RM26CDNYYHC3Q4IXAPA5CNFSM4KC4WASKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IEC7XVA&data=02%7C01%7CD.Lakens%40tue.nl%7Cf09fe977d0874746bc8608d792140f02%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637138487176939498&sdata=jNnIY0NVJ%2Becg%2FLanP6%2F%2Bm7DRQnK3qMRYcjL0Byd55I%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABRQPYKZ2AQEIWNBOJ2BK33Q4IXAPANCNFSM4KC4WASA&data=02%7C01%7CD.Lakens%40tue.nl%7Cf09fe977d0874746bc8608d792140f02%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637138487176949492&sdata=or55Yt%2BK4epIZZZME7joSoPsU1AQ3%2B2WnbvymJ3Wleo%3D&reserved=0.

amscheel commented 4 years ago

Hi Heather, I just checked this with the supporting files and code we shared with the manuscript (https://github.com/Lakens/EquivalenceTestingForPsychologicalScience) and I get the same result as the one we report, so I don't think it's a TOSTER version issue. Looking at the values I think the difference is probably due to the bound (you're using .25 and we're using .2511, although your proportions are also truncated compared to ours, though those probably cause more minor differences). Hope this helps!

TOST results: Z-value lower bound: 5.27 p-value lower bound: 0.00000007 Z-value upper bound: -9.12 p-value upper bound: 0.00000000000000000004

Equivalence bounds: low eqbound: -0.2511 high eqbound: 0.2511

TOST confidence interval: lower bound 90% CI: -0.125 upper bound 90% CI: -0.01

NHST confidence interval: lower bound 95% CI: -0.136 upper bound 95% CI: 0.001

Equivalence Test based on Fisher's exact z-test Result: The equivalence test was significant, Z = 5.274, p = 0.0000000667, given equivalence bounds of -0.251 and 0.251 and an alpha of 0.05.

Null-Hypothesis Fisher's exact z-test Result: The null hypothesis test was non-significant, Z = -1.925, p = 0.0542, given an alpha of 0.05.

Based on the equivalence test and the null-hypothesis test combined, we can conclude that the observed effect is statistically not different from zero and statistically equivalent to zero.

HeatherUrry commented 4 years ago

Hi there - thanks so much for responding and for pointing out that OSF material. Very helpful.

The difference is that your code specifies +/- .25 as equivalence bounds and reports the Z for the upper bound, which is -9.12. My code specifies -1 and .25 for lower and upper bounds, respectively. Hence the Z of -.9094. The difference is negligible, of course - I just wanted to understand!

Heather

On Sun, Jan 5, 2020 at 2:27 PM Daniel Lakens notifications@github.com wrote:

Hi, the manuscript you took the example from has fully reproducible code in R markdown. Can you please take a look at the code linked to in the disclosures? What happens if you take the test from our manuscript and compare it to the code you created?

From: HeatherUrry notifications@github.com Sent: Sunday, January 5, 2020 8:18:31 PM To: Lakens/TOSTER TOSTER@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [Lakens/TOSTER] TOSTtwo.prop question (#48)

Hi Daniël,

As part of my attempt to get smarter about inferiority tests, I'm trying to replicate Example 4 from Lakens, Scheel, & Isager (2018), https://journals.sagepub.com/doi/pdf/10.1177/2515245918770963< https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjournals.sagepub.com%2Fdoi%2Fpdf%2F10.1177%2F2515245918770963&data=02%7C01%7CD.Lakens%40tue.nl%7Cf09fe977d0874746bc8608d792140f02%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637138487176929503&sdata=KYJb50MiBAvklyrF2yJwxChJ0YFCctZJka92zkCEMVc%3D&reserved=0> using TOSTER.

The text says, "The inferiority test on the data from Lynott et al. reveals that one can reject effects larger than Δ = 0.25, z = –9.12, p < .001 (see Fig. 2d)" (p. 266).

But when I run the TOSTtwo.prop command (below), it gives a result of Z = -9.094, p < .001. I thought it might be rounding error, but that doesn't seem to be the case. I downloaded Lynott et al.'s data and calculated the proportions to 7 decimal places. What am I doing wrong? Alternatively, does this perhaps reflect a version issue? I'm using TOSTER v 0.3.4.

Thanks! (And thanks too for writing this package. It's really useful.)

Heather

TOSTtwo.prop(prop1 = 0.5074257, prop2 = 0.5745721, n1 = 404, n2 = 409 , low_eqbound = -1, high_eqbound = .25, alpha =.05)

TOST results: Z-value lower bound: 26.75 p-value lower bound: 0.000 Z-value upper bound: -9.09 p-value upper bound: 0.00000000000000000005

Equivalence bounds: low eqbound: -1 high eqbound: 0.25

TOST confidence interval: lower bound 90% CI: -0.125 upper bound 90% CI: -0.01

NHST confidence interval: lower bound 95% CI: -0.136 upper bound 95% CI: 0.001

Equivalence Test based on Fisher's exact z-test Result: The equivalence test was significant, Z = -9.094, p = 0.0000000000000000000479, given equivalence bounds of -1.000 and 0.250 and an alpha of 0.05.

Null-Hypothesis Fisher's exact z-test Result: The null hypothesis test was non-significant, Z = -1.925, p = 0.0542, given an alpha of 0.05.

Based on the equivalence test and the null-hypothesis test combined, we can conclude that the observed effect is statistically not different from zero and statistically equivalent to zero.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FLakens%2FTOSTER%2Fissues%2F48%3Femail_source%3Dnotifications%26email_token%3DABRQPYIBJQ3RM26CDNYYHC3Q4IXAPA5CNFSM4KC4WASKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IEC7XVA&data=02%7C01%7CD.Lakens%40tue.nl%7Cf09fe977d0874746bc8608d792140f02%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637138487176939498&sdata=jNnIY0NVJ%2Becg%2FLanP6%2F%2Bm7DRQnK3qMRYcjL0Byd55I%3D&reserved=0>, or unsubscribe< https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABRQPYKZ2AQEIWNBOJ2BK33Q4IXAPANCNFSM4KC4WASA&data=02%7C01%7CD.Lakens%40tue.nl%7Cf09fe977d0874746bc8608d792140f02%7Ccc7df24760ce4a0f9d75704cf60efc64%7C1%7C0%7C637138487176949492&sdata=or55Yt%2BK4epIZZZME7joSoPsU1AQ3%2B2WnbvymJ3Wleo%3D&reserved=0>.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Lakens/TOSTER/issues/48?email_source=notifications&email_token=AHKOXLSRUAWVIPDLJO64NATQ4IYCZA5CNFSM4KC4WASKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEID5TYA#issuecomment-570939872, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKOXLXFRK22RPXPZHNBKITQ4IYCZANCNFSM4KC4WASA .

-- Heather L. Urry, Ph.D. Department of Psychology Tufts University 490 Boston Avenue Medford, MA 02155

pronouns: she/her/hers

email: heather.urry@tufts.edu phone: 617-627-3733 fax: 617-627-3181

http://ase.tufts.edu/psychology/ebbl @HeatherUrry https://twitter.com/HeatherUrry

HeatherUrry commented 4 years ago

Ah! The .2511 vs .25 difference must be the answer indeed. Thanks, Anne!

H.

On Sun, Jan 5, 2020 at 2:40 PM amscheel notifications@github.com wrote:

Hi Heather, I just checked this with the supporting files and code we shared with the manuscript ( https://github.com/Lakens/EquivalenceTestingForPsychologicalScience) and I get the same result as the one we report, so I don't think it's a TOSTER version issue. Looking at the values I think the difference is probably due to the bound (you're using .25 and we're using .2511, although your proportions are also truncated compared to ours, though those probably cause more minor differences). Hope this helps!

TOST results: Z-value lower bound: 5.27 p-value lower bound: 0.00000007 Z-value upper bound: -9.12 p-value upper bound: 0.00000000000000000004

Equivalence bounds: low eqbound: -0.2511 high eqbound: 0.2511

TOST confidence interval: lower bound 90% CI: -0.125 upper bound 90% CI: -0.01

NHST confidence interval: lower bound 95% CI: -0.136 upper bound 95% CI: 0.001

Equivalence Test based on Fisher's exact z-test Result: The equivalence test was significant, Z = 5.274, p = 0.0000000667, given equivalence bounds of -0.251 and 0.251 and an alpha of 0.05.

Null-Hypothesis Fisher's exact z-test Result: The null hypothesis test was non-significant, Z = -1.925, p = 0.0542, given an alpha of 0.05.

Based on the equivalence test and the null-hypothesis test combined, we can conclude that the observed effect is statistically not different from zero and statistically equivalent to zero.

—

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Lakens/TOSTER/issues/48?email_source=notifications&email_token=AHKOXLWHMSJPQI7FACM3V2DQ4IZSTA5CNFSM4KC4WASKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEID53LY#issuecomment-570940847, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKOXLULCBIHD373SMZ5ZHLQ4IZSTANCNFSM4KC4WASA .

-- Heather L. Urry, Ph.D. Department of Psychology Tufts University 490 Boston Avenue Medford, MA 02155

pronouns: she/her/hers

email: heather.urry@tufts.edu phone: 617-627-3733 fax: 617-627-3181

http://ase.tufts.edu/psychology/ebbl @HeatherUrry https://twitter.com/HeatherUrry