Easier power calculations

[SaskiaFreytag]

It would be great to just provide a fitted model of a pilot and its data and one could get a power analysis back.

[Lingtax]

Are you meaning like a monte carlo power analysis based on the observed effects? Do we need to be wary of using pilots for that? My understanding is that this might lead to underpowered designs.

[SaskiaFreytag]

No I mean a way simpler analysis, where you fit a model of some pilot data and then feed it into a power calculator with little frills. I have not thought about the issues around the use of pilots. While I understand that there may be problems with the use of pilots, I think there is currently no alternative for at least controlled animal experiments but the use of pilots.

[Lingtax]

I think I am misunderstanding your process a little.

I've built a shiny dashboard around the pwr package for psych studies are you thinking something that might do something similar, but with different inputs?

[SaskiaFreytag]

I think the input to a power calculation should just be the model fit. eg.

mod<- lm(y~x1+x2,data=dat) our.power.function(mod,"x1",a=0.05,pow=0.8)

you need 15 samples

[Lingtax]

And this is where x1 and x2 are factors I presume? I think I can see how we can do this, leveraging pwr still. 😄

Mostly for discussion's sake, in psych the movement now is drifting from pilots for effect size estimation to anchoring power on something closer to Minimal Important Difference.

[fsingletonthorn]

My PhD thesis is on this! Happy to chat about this issue. See the attached document for a (as yet not properly proof read) paper on the implications of different methods of power analysis / effect size selection. It's basically the "Why did no one explain the necessary-conceptual-starting-points for power analysis in way I could understand before I started writing a PhD on power analysis" type paper / chapter. Approaches to formal sample size determination.docx

[SaskiaFreytag]

That is really cool, thanks for sharing. However, I still think that it very much depends on your field of research whether alternative approaches to a power analysis (i.e. minimum effect size that we care about) are practical. When you consider animal experiments this minimum effect size might just not be known/ or very small and in this case using estimates from pilot studies seem legitimate.

[SaskiaFreytag]

Also I just saw that you discuss that very nicely.

[fsingletonthorn]

Thanks! There's no easy solution I think. But I'd be more than happy to work on tools for making sample size determination / effect size estimation easier & more intuitive. The idea that I've been thinking about for the last few years or so is making a general purpose package for power analysis through simulation ~ there are one or two specific packages that I know about, but nothing particularly general / where you could do some more idiosyncratic analyses with known assumption violations. It's a big project though and making it any easier than what is already available is difficult. The biggest thing would be to build a SEM diagram / DAG UI for specifying the data generating process, but that's well beyond my abilities at this point.

[fsingletonthorn]

The other thing that we could plausibly build using model objects would be a package to perform assurance-style power analysis using a prior distribution based on the observed data w/ a flat prior,i.e., estimate the probability that they will get statistically significant effect given the observed data. Which, I think, is what people might sort of think they get when they do a power analysis on pilot data.

[egouldo]

Just wanted to share these two examples from ecological modelling applications. They're bespoke approaches, but there might be some useful bits in there.

1. Thomas et al. (2018) A field ecologist's adventures in the virtual world: using simulations to design data collection for complex models. Ecological Applications. https://esajournals.onlinelibrary.wiley.com/doi/abs/10.1002/eap.1801 A "virtual ecologist simulation" approach to designing field-work studies, also has a GitHub repo of the code: https://doi.org/10.5281/zenodo.1400861
2. Laubmeier et al. (2018) From theory to experimental design— Quantifying a trait-based theory of predator- prey dynamics. PLOSone. https://doi.org/10.1371/journal.pone.0195919 Paper looks at how to optimally design the experiment to best test the validity of some ecological theory

[tslumley]

Also, have a look at Peter Green's SIMR package, which does power for GLMMs. https://github.com/pitakakariki/simr https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12504