Psy2R - an R package for better inference in multivariate statistical analysis

Title

Short description and the goals for the OHBM BrainHack

We consistently use massive datasets across neuroscience and psychology. The routine gathering of big data requires that we are well equipped with tools that allow us to conduct appropriate multivariate statistics. This project aims to produce an R package that allows the researcher to overcome little discussed limitations of traditional multivariate statistical analyses.

Multivariate statistical analysis (e.g. MANOVA and repeated-measures ANOVA) typically follows a two stage procedure; an omnibus test of the global null hypothesis followed by post-hoc tests of specific effects. It is not well known that under certain circumstances, such as when the omnibus test is overpowered, that the type 1 error rate for this procedure is drastically inflated, sometimes to a type 1 error rate of 1! It is even less well known that this procedure can lead to an even lessor known type IV error, which is the incorrect interpretation of a correctly rejected hypothesis. This is caused when the follow-up contrasts are inadequate to test the question of interest, as can occur when testing simple effects.

It is possible to avoid these dragons by using an alternative procedure where all inferences are derived from simultaneous confidence intervals (SCIs) on contrasts of interests. The 'simultaneous' bit means that the same statistic contributes to both the omnibus and the contrast tests for significance, which controls the type 1 error rate. Even better, computing confidence intervals on contrasts of interests allows reseachers to move away from binary decision-making (is something significant or not?) to interpretations involving magnitude (how big is this effect likely to be at the population level?).

One piece of software (PSY) can produce SCIs appropriate for both planned analyses (where contrasts are defined independently of the data) and for more flexible analyses where contrasts are defined on a post-hoc basis. However, this software is only available for use on windows and cannot be scripted into reproducible workflows. Our goal is to build an R package that implements the functions of PSY, and to make this method of statistical inference available to the masses!

Our goals for OHBM Brainhack 2024 are:

Convert some key functions of the Psy source pascal code to R functions: these functions compute the largest contrast effect you could expect to get if the null hypothesis is true, when you have between and within repeated measures.
Explore the overlap between Psy, emmeans and MBESS, to make sure we recycle where appropriate
Replicate analyses between the original Psy software and the R implementation

Link to the Project

https://github.com/kel-github/PSY2R

Image/Logo for the OHBM brainhack website

https://github.com/kel-github/PSY2R/blob/main/presentations/Psy2R-logo.jpeg

Project lead

Kelly Garner Github: kel-github Discord: @kel-accords

Main Hub

Seoul

Link to the Project pitch

No response

Other hubs covered by the leaders

[X] Seoul
[ ] Hybrid (Asia / Pacific)
[ ] Hybrid (Europe / Middle East / Africa)
[ ] Hybrid (Americas)

Skills

Coding skills in R Package development in R Understanding of, experience in, or curiosity with multivariate statistical analysis Understanding of, experience in, or curiosity with analysis of repeated-measures data (i.e. ANOVA+) A desire for tools for better statistical inferences A love of documentation and/or sensible naming of variables Esoteric coding skills - i.e. pascal

Good first issues

https://github.com/kel-github/PSY2R/issues/33 https://github.com/kel-github/PSY2R/issues/34 https://github.com/kel-github/PSY2R/issues/12

Twitter summary

Psy2R - an R package for better inference in multivariate statistical analysis. We're making a package to reduce type 1 and type 4 errors in multivariate analysis - for the people!

Short name for the Discord chat channel (~15 chars)

Psy2R

Please read and follow the OHBM Code of Conduct

[X] I agree to follow the OHBM Code of Conduct during the hackathon

ohbm / hackathon2024