search

rasbt / mlxtend

A library of extension and helper modules for Python's data analysis and machine learning libraries.
https://rasbt.github.io/mlxtend/
Other
4.86k stars 857 forks source link

Independent t-test #807

Open kaz94 opened 3 years ago

kaz94 commented 3 years ago

Comparing the same algorithm but on different data sets, and deciding whether the difference in a chosen metric is statistically significant or just due to randomness could be useful.

rasbt commented 3 years ago

It might be useful to add a 2-sample independent t-test in general, but I think using either (a) a 5x2cv paired t test or McNemar test is better for classifier comparisons

EDIT: I see you are asking for the different-dataset scenario. Yes, then these two above cannot be used and either a permutation test of 2-sample independent t-test may be used.

kaz94 commented 3 years ago

Yes, I suggested it since I needed to use the independent version to compare the same classifier but on different data sets :)

niedz., 14 mar 2021, 16:57 użytkownik Sebastian Raschka < @.***> napisał:

It might be useful to at 2-sample independent t-test in general, but I think using either (a) a 5x2cv paired t test or McNemar test is better for classifier comparisons

- http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rasbt/mlxtend/issues/807#issuecomment-798931391, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEGSVABKACEU2IP6SDJ7T63TDTMHZANCNFSM4Y7C2DCA .