maximtrp / scikit-posthocs

Multiple Pairwise Comparisons (Post Hoc) Tests in Python
https://scikit-posthocs.rtfd.io
MIT License
346 stars 40 forks source link

[DOC, MAINT] Small fixes, unit tests and docs for CD diagrams #58

Closed pedroilidio closed 1 year ago

pedroilidio commented 1 year ago

Includes the new CD diagram feature added by https://github.com/maximtrp/scikit-posthocs/pull/57 in the documentation. The plots were generated with the following script:

gen_tutorial_critical_difference_figures.py ```python import numpy as np import pandas as pd import scipy.stats as ss import matplotlib.pyplot as plt import scikit_posthocs as sp # First example rng = np.random.default_rng(1) dict_data = { "Sample 1": rng.normal(loc=0.15, scale=0.1, size=30), "Sample 2": rng.normal(loc=0.2, scale=0.1, size=30), "Sample 3": rng.normal(loc=0.2, scale=0.1, size=30), "Sample 5": rng.normal(loc=0.3, scale=0.1, size=30), "Sample 6": rng.normal(loc=0.51, scale=0.1, size=30), "Sample 7": rng.normal(loc=0.7, scale=0.1, size=30), "Sample 8": rng.normal(loc=0.7, scale=0.1, size=30), "Sample 9": rng.normal(loc=0.8, scale=0.1, size=30), "Sample 10": rng.normal(loc=0.82, scale=0.1, size=30), "Sample 12": rng.normal(loc=0.9, scale=0.1, size=30), "Sample 4": rng.normal(loc=0.25, scale=0.1, size=30), "Sample 10": rng.normal(loc=0.79, scale=0.1, size=30), "Sample 11": rng.normal(loc=0.84, scale=0.1, size=30), } data = ( pd.DataFrame(dict_data) .rename_axis("block") .melt( var_name="sample", value_name="score", ignore_index=False, ) .reset_index() ) avg_rank = data.groupby("block").score.rank(pct=True).groupby(data["sample"]).mean() test_results = sp.posthoc_conover_friedman( data, melted=True, block_col="block", group_col="sample", y_col="score" ) plt.figure(figsize=(8, 2)) plt.title("Example of Critical Difference Diagram") sp.critical_difference_diagram( avg_rank, test_results, label_props={"fontweight": "bold"} ) plt.tight_layout() plt.savefig("docs/source/_static/cd_diagram0.png", bbox_inches="tight") # Second diagram rng = np.random.default_rng(1) dict_data = { "model1": rng.normal(loc=0.2, scale=0.1, size=30), "model2": rng.normal(loc=0.2, scale=0.1, size=30), "model3": rng.normal(loc=0.4, scale=0.1, size=30), "model4": rng.normal(loc=0.5, scale=0.1, size=30), "model5": rng.normal(loc=0.7, scale=0.1, size=30), "model6": rng.normal(loc=0.7, scale=0.1, size=30), "model7": rng.normal(loc=0.8, scale=0.1, size=30), "model8": rng.normal(loc=0.9, scale=0.1, size=30), } data = ( pd.DataFrame(dict_data) .rename_axis("cv_fold") .melt( var_name="estimator", value_name="score", ignore_index=False, ) .reset_index() ) ss.friedmanchisquare(*dict_data.values()) avg_rank = data.groupby("cv_fold").score.rank(pct=True).groupby(data.estimator).mean() test_results = sp.posthoc_conover_friedman( data, melted=True, block_col="cv_fold", group_col="estimator", y_col="score" ) # Significance heatmap for the example plt.figure() sp.sign_plot(test_results) plt.savefig("docs/source/_static/cd_diagram_example_sig_plot.png") # Third diagram plt.figure(figsize=(8, 2)) plt.title("Critical difference diagram of average score ranks") sp.critical_difference_diagram(avg_rank, test_results) plt.tight_layout() plt.savefig("docs/source/_static/cd_diagram1.png") # Fourth and final, stylized diagram plt.figure(figsize=(8, 2)) plt.title("Critical difference diagram of average score ranks") sp.critical_difference_diagram( ranks=avg_rank, sig_matrix=test_results, label_fmt_left="{label} [{rank:.3f}] ", label_fmt_right=" [{rank:.3f}] {label}", text_h_margin=0.3, label_props={"color": "black", "fontweight": "bold"}, crossbar_props={"color": None, "marker": "o"}, marker_props={"marker": "*", "s": 150, "color": "y", "edgecolor": "k"}, elbow_props={"color": "gray"}, ) plt.tight_layout() plt.savefig("docs/source/_static/cd_diagram2.png") ```

I'll soon be adding some unit tests in another PR.

I ended up adding the tests and some fixes to this same branch, I hope it doesn't get too cluttered to revise.

Here is a summary of the changes:

Please let me know if there are any other modifications or additions that would be interesting.

Edit: spacing and new commits description.

maximtrp commented 1 year ago

Nice! Thank you!

pedroilidio commented 1 year ago

You are welcome! I'm glad to contribute to this work!