search

treder / MVPA-Light

Matlab toolbox for classification and regression of multi-dimensional data
MIT License
70 stars 35 forks source link

clustercritval for AUC? #45

Open yoojchoo opened 7 months ago

yoojchoo commented 7 months ago

Hi, I've been working through the tutorials in MVPA Light toolbox. As I'm using the AUC and am interested in whether the decoding performance is above chance level (0.5), I was thinking that I was going to use the critical cut-off z =1.65 (right-tailed). However, in the tutorial, the z val is set as 1.96 which corresponds to uncorrected p value of 0.05 (two-tailed).

Could you explain why it is the case?

Thank you in advance!

treder commented 6 months ago

Hi @yoojchoo ! Thank you for raising a good point. In principle there's no correct or incorrect choice, it all depends on what you want to achieve. I agree that you can do a right-tailed test in most cases and added some more documentation to the tutorial:

+% Note that these z-values correspond to a two-sided test. In many cases it
+% suffices to use a right-sided test. In this case, the following cutoff
+% values are obtained:
+% z-val = 1.282 corresponds to uncorrected p-value = 0.1
+% z-val = 1.645 corresponds to uncorrected p-value = 0.05
+% z-val = 2.326 corresponds to uncorrected p-value = 0.01

Two-sided tests can be useful when there's a possibility of a pattern-reversal between train and test data (e.g., in memory experiments).

yoojchoo commented 6 months ago

Hi Dr. Treder, Thanks for your reply! The reason I was confused was that the MVPA-Light toolbox has only one-sided test (am I right?) but the Z value was set (at least in the tutorial) as 1.96. But I see your point.

Following this, I'm wondering if you are planning to add two-sided test in your toolbox at some point.

Again, thanks for the great toolbox!

Best, Yoojeong

treder commented 6 months ago

Yes, you are right, it's one-sided tests (lower or upper tail). In principle a two-tailed test could be implemented. The problem is that a two-tailed test mixes together both directions (significantly higher than chance and significantly lower than chance), although they represent different types of effect. It's numerically possible but I'm not sure it's useful. It might be better to run both test (tail = 1 and tail = -1) separately.

Thank you for the feedback, glad you find MVPA-Light useful :)