[Feature Request]: Sensitivy for observed effects

[Trebicky]

Description

Each (frequencists) test should provide Reliability of given result or Sensitivy to obsreve it.

Purpose

Provide users with measur of theyr analysis sensitivy or reliability. It would help user to make better conclusions about obsreved effects. Such feature would be singlehandedly be the biggest improvent in result intepretation any software currently provide - though StatsCloud.app does a similar thing with no setting available.

Use-case

Suppose I would run a bivariate correlation analysis (with Pearson's r, but it shoudl not mattre what effect size is used), at taht point JASP does know N in the test and does know the obsreved r. It could only provide additional column (line in cell) specifing sensitivy to observe such r. In analysis setting user can specify type 1 error rate or use some default (e.g. 0.05). Maybe user can specifiy also their Beta rate and see either a) whether the N was big enough or not, b) what power they had to observe such effect, c) what effect is the smallest they can observed with their N reliabily. This could be implemented in was majority of Freq test (parametric and non-para) available in JASP, including meta-analyses.

Is your feature request related to a problem?

People still tend to interpret test results regardless power to observe them. Current tools are insuficient in UX. JASP does have potentila to chage the game.

Describe the solution you would like

Each test would report sensitivity to observe given effect and addition setting to specify criteria of alpha and beta rates, maybe a power curve plot.

Describe alternatives that you have considered

Separate power analysis module featuring a priory, sensitivy, presicion estimates test

Additional context

No response

[Kucharssim]

Dear @Trebicky,

We are planning to add a module to calculate power for typical statistical tests.

As for the actual feature you are proposing, I am not sure but I seems to me like calculating the "observed power" after analysing the data. As far as I know, there is a consensus that this statistic does not offer any information above the p-value and in general should not be used (e.g., https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf). Perhaps @EJWagenmakers, @AlexanderLyNL would have some opinion about this feature?

[EJWagenmakers]

Yes, observed power is a transformation of the p-value and statisticians generally get upset whenever this concept is mentioned :-)

[Trebicky]

Thanks for your answer. I am aware of post hoc power nono, but running a sensitive analysis should be deem ok. Thus I suggest only to compare observed effect with effect for which given sample and analysis is suffciently poweredr for.

Dne út 2. 8. 2022 11:33 uživatel EJ @.***> napsal:

Yes, observed power is a transformation of the p-value and statisticians generally get upset whenever this concept is mentioned :-)

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/1792#issuecomment-1202250003, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANG47BBE6H2GU4BUA25GOILVXDTO3ANCNFSM55KFLJ2Q . You are receiving this because you were mentioned.Message ID: @.***>

[tomtomme]

We have the power module in, even the post hoc part. I think post hoc power is informative for future research like replications. So thats great :D Even if it is a transormation of the p-value, it is easier to interpret in the context of "How much power did I have in retrospect and how beneficial would it be to increase n the next time according to my power curve assuming the mean effect stays the same over many more replications"

Now @Trebicky proposed a sensitivity analysis on top - integrated into every classic test. That would be even more helpful - maybe as a long term goal? Possible much easier is to prioritize the inclusion of more tests into the existing power module starting with regressions and ANOVAs. @jansim Are you working on an extension?

[Trebicky]

Congratulation on the new Power module. It is a feature I was very excited about being added to Jasp.

Though reporting sensitivity alongside classical tests would be my favourite feature, I strongly agree that a much-needed step forward would be a richer module with more power analyses.

I am glad what is now in the first version, and I hope the newer version will include even more common tests like bi-variate correlations, ANOVAs (RM, one and multiple ways), linear regressions, X2 tests, meta-analyses etc. - in sum, all the tests, that can be run in jasp.

This leads me to several points I would like to raise for potential future improvements/creature comfort.

• Currently, the module contains "only" a few tests without much structure. When the module is richer, it would be easier to navigate if it were structured similarly to modules in Jasp - e.g. T-test, Anovas, Regression, Frequencies, Meta-analyses...
• This leads to a feeling that the current Power module is now designed more as a teaching tool than a work tool for analyses (in contrast to ubiquitous but less straightforward G*power). It would be nice to tie the Power and other analysis modules together (like labels for effect sizes, the same effects used in Power and results, etc...)
• In the case of pair samples t-test, jasp does not require the correlation between the two repeated measures. However, it is frequently used in repeated measure power elsewhere. Could you add this as an option to fill in?
• Currently, one can run traditional power analyses for (a priori) sample size, (sensitivity) effect size, (post-hoc) power - great!. Could you also add precision? So I could estimate the sample size needed for particular width of CI around the mean effect size (e.g. not containing O, not being lower than XYZ) as suggested by Dienes or Lakens?

Thank you for the great work!

[EJWagenmakers]

Yes, this is definitely a first step, and we hope to extend the functionality in the future along the lines you suggest. E.J.