Psychonomics 2020: Power simulations for linguistic norm data collection

[doomlab]

Type: Talk

Title: Power simulations for linguistic norm data collection

Session types: Language: semantics, Language: psycholinguistics

Authors:

• Erin M. Buchanan, ebuchanan@harrisburgu.edu, Harrisburg University of Science and Technology
• K.D. Valentine, kvalentine2@mgh.harvard.edu, Massachusetts General Hospital
• Nicholas Maxwell, nicholas.maxwell@usm.edu, University of Southern Mississippi
• Jack E. Taylor, Jack.Taylor@glasgow.ac.uk, University of Glasgow
• Maria Montefinese, maria.montefinese@gmail.com, University of Padova
• (others as added by their approval)

Abstract (1250 characters max):

The current focus on replication and reproducibility has driven a need to adequately power studies through appropriate sample size planning. However, estimating power and sample size is usually driven by the choice of hypothesis test and research design. An incredible number of psycholinguistic database norms have been published in the last ten years, and the issues of power and sample size have largely been ignored, as these studies do not use hypothesis testing as a main objective. This presentation will discuss how to use accuracy in parameter estimation (AIPE) and qualitative coverage approaches to determine the appropriate number of participants for data collection in a study with no hypothesis test. Data from English feature production norms, the English Lexicon Project, and participant ratings (i.e., valence, concreteness, etc.) will be used to demonstrate how to estimate variable sample sizes by item for both qualitative (feature production norms) and quantitative (priming, lexical decision tasks, judgment tasks) type data.

[doomlab]

Updated with KD and Nick as authors (Felix declined for this presentation).

[JackEdTaylor]

This sounds cool! I've never done this kind of power analysis so I have a few questions:

• What's qualitative coverage?
• To check I'm on the right wavelength, will the AIPE find the number of subjects needed to reach a suitably small confidence interval/standard error of parameters from a model fit to the data?
• Since AIPE power will differ by effect size, could it make sense to show how you can calculate the power of your norms for accurately estimating different effect sizes (e.g. plot overlaid power curves for effect sizes from 0.1 to 1)?

[doomlab]

• Coverage is the term for making sure a qualitative analysis adequately samples the problem space. So, if you are trying to find themes or topics, you might use a coverage approach to make sure you've gotten them all. There's a new paper out on how to determine sample size for property norms (reviewed it, coming soon to BRM), but here's an earlier paper in their line of work conceptualizing the idea of variability: https://link.springer.com/article/10.3758/s13428-017-0920-8
• You are correct in your understanding of AIPE - I have the simulations we did for the PSA proposal to estimate appropriate sample size for each item by finding a "sufficiently narrow" confidence interval, which ends up being a target standard error to shoot for.
• On this third point, the trouble here is that there is no effect size, since you aren't doing any type of test. Take RT for example - we aren't interested in how different they are from each other, how different they are from zero, or anything except that they are measured well (which is defined as having enough people the confidence interval stabilizes and is small). So, I'm unsure how that would influence effect size, except that the SD or SE as the denominator of the effect size or test statistic would be less variable. If we are shooting for a specific SE then doesn't that influence them in the same way? (i.e., bigger effects are bigger because the numerator of the effect size is large, and we are holding the denominator constant). I could be wrong though if you have some visualization ideas.
• PS, adding you to the author list, good thoughts :)

[MarMon83]

Hi Erin, thank you so much for putting this together! The abstract sounds really great! I was wondering if it could be possible to extend AIPE to rating norms. If I think affective ratings (like valence, arousal, dominance), I'm not sure it could be possible given the expected high variability in participants' judgments. But, what do you think about lexical-semantic measures, like concreteness, mode of acquisition, imageability and so forth, which seem more "objective" measures (so with less individual variability)?

[doomlab]

@MarMon83 - great question - definitely we can add it to the paper, and I'll add it to the presentation too. I'm unsure if one of those is more difficult than the other - mainly, I think the confidence interval or SE would be a larger for the more variable ones, but that's a good question we can test in the paper.