NHST? or revise...

[mcfrank]

From Steve Goodman:

I do have some cavils re your descriptions of what you call Fisherian and N-P NHST. I have attached some things I and others have written that that bear on this, but the bottom lines are below.

• [x] The approaches of Fisher and N-P are dramatically different, and should not have the same name. By using “NHST” to label both approaches, you obscure the profound differences between them, making them easier to conflate– what Gigererenzer called the “silent hybrid solution.” (attached, p. 106) I would use terminology and a description that much draws a much clearer line between the two; Fisher’s approach was distinctly evidential, and was barely “testing,” b/c further data gathering was always what was intended. You capture this in a footnote, where you say that Fisher was in favor of replication, but this is not a footnote feature; it was central to his approach to inference.
• [x] Continuing on the same theme, N-P hypothesis tests do not use p-values. The calculation of a p-value as a part of hypothesis testing was something Fisher, Neyman and Pearson all objected to, and is a cobbled-together, incoherent feature of the approach we now use.
• [x] Your invocation of Jeffreys and the descriptors of the BF calibration are good, but that was developed mostly with physics in mind and proposed in 1935. More modern descriptors and scaling are in writings of Jim Berger, Richard Royall, myself, and I am sure EJ Wagenmakers and Alex Etz. See Tables 1 & 2 in the attached Annals piece; what Jeffreys labels as “Substantial” is better described as “Moderate”, which has a very different connotation. (Do we really want to describe a p=0.07 as providing “substantial” evidence?)
• [x] The year for the 3d Edition of Jeffreys book is 1961, not 1998 (which was the date of the Oxford paperback re-publication, but not the actual copyright date of this edition). The citation should also specify 3rd edition, as the first was 1939.

[mcfrank]

@mayamathur I added a footnote on this, cited Steve's standards, and tried to clarify some of the N-P stuff a little. You are welcome to review if you want.

[mayamathur]

Will review soon!

On Dec 18, 2023, at 1:49 PM, Michael Frank @.***> wrote:

Closed #208 https://github.com/langcog/experimentology/issues/208 as completed via 80e345f https://github.com/langcog/experimentology/commit/80e345f3308e0bd3f8e535d315df822741f048a3.

— Reply to this email directly, view it on GitHub https://github.com/langcog/experimentology/issues/208#event-11282899215, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEE6HWPVLXXRXDTKKFEUYDYKDJBHAVCNFSM6AAAAAA6EMEIM2VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGI4DEOBZHEZDCNI. You are receiving this because you were mentioned.