AmenRa / ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍
https://amenra.github.io/ranx
MIT License
427 stars 23 forks source link

[Question] About the correction among multiple hypotheses #61

Closed striver314 closed 1 month ago

striver314 commented 7 months ago

I came across a point in the Fuhr article where it talks about the need for additional correction for multiple hypotheses test (e.g., Bonferroni correction). Looking at the paired_student_t_test function code, it seems like the correction procedure is not taken into account.

Am I on the right track with this? If the current version does indeed consider this, could you guide me on how to verify it?

AmenRa commented 1 month ago

Hi and sorry for the delay.

You can apply Bonferroni correction by changing the max_p parameter of the compare function (Maximum p-value to consider an increment as statistically significant. Defaults to 0.01.).

Otherwise, you can use the Tukey's HSD Test, as suggested by Fuhr in the same article, by passing Tukey as stat_test parameter of compare.

Closing.