I'll soon be adding some unit tests in another PR.
I ended up adding the tests and some fixes to this same branch, I hope it doesn't get too cluttered to revise.
Here is a summary of the changes:
[DOC] A section is added to the tutorial briefly describing critical difference diagrams.
[FIX] pyplot.annotate() was replaced by pyplot.text() in the plotter function. The former is sometimes not included in the figure bounding area when setting transparency or using tight_layout(), for example.
[DOC, FIX] Improved function implementing Bron-Kerbosch algorithm to find crossbars for critical difference diagrams. Now simple input validation is performed and it does not enters infinite recursion when empty adjacency matrices are received.
[DOC] Docs for the aforementioned function and variable names also have improved readability.
[ENH] CD diagrams now plots the crossbars from the left-most position, keeping them more organized and consistent across runs.
[FIX] Previously, if a sample was different from all the others, a crossbar contained only it was still being created, and visible if
a marker was set in crossbar_props. critical_differece_diagram now requires at least two samples to plot a crossbar.
[MAINT] Some tests for CD diagrams were implemented.
Please let me know if there are any other modifications or additions that would be interesting.
Includes the new CD diagram feature added by https://github.com/maximtrp/scikit-posthocs/pull/57 in the documentation. The plots were generated with the following script:
gen_tutorial_critical_difference_figures.py
```python import numpy as np import pandas as pd import scipy.stats as ss import matplotlib.pyplot as plt import scikit_posthocs as sp # First example rng = np.random.default_rng(1) dict_data = { "Sample 1": rng.normal(loc=0.15, scale=0.1, size=30), "Sample 2": rng.normal(loc=0.2, scale=0.1, size=30), "Sample 3": rng.normal(loc=0.2, scale=0.1, size=30), "Sample 5": rng.normal(loc=0.3, scale=0.1, size=30), "Sample 6": rng.normal(loc=0.51, scale=0.1, size=30), "Sample 7": rng.normal(loc=0.7, scale=0.1, size=30), "Sample 8": rng.normal(loc=0.7, scale=0.1, size=30), "Sample 9": rng.normal(loc=0.8, scale=0.1, size=30), "Sample 10": rng.normal(loc=0.82, scale=0.1, size=30), "Sample 12": rng.normal(loc=0.9, scale=0.1, size=30), "Sample 4": rng.normal(loc=0.25, scale=0.1, size=30), "Sample 10": rng.normal(loc=0.79, scale=0.1, size=30), "Sample 11": rng.normal(loc=0.84, scale=0.1, size=30), } data = ( pd.DataFrame(dict_data) .rename_axis("block") .melt( var_name="sample", value_name="score", ignore_index=False, ) .reset_index() ) avg_rank = data.groupby("block").score.rank(pct=True).groupby(data["sample"]).mean() test_results = sp.posthoc_conover_friedman( data, melted=True, block_col="block", group_col="sample", y_col="score" ) plt.figure(figsize=(8, 2)) plt.title("Example of Critical Difference Diagram") sp.critical_difference_diagram( avg_rank, test_results, label_props={"fontweight": "bold"} ) plt.tight_layout() plt.savefig("docs/source/_static/cd_diagram0.png", bbox_inches="tight") # Second diagram rng = np.random.default_rng(1) dict_data = { "model1": rng.normal(loc=0.2, scale=0.1, size=30), "model2": rng.normal(loc=0.2, scale=0.1, size=30), "model3": rng.normal(loc=0.4, scale=0.1, size=30), "model4": rng.normal(loc=0.5, scale=0.1, size=30), "model5": rng.normal(loc=0.7, scale=0.1, size=30), "model6": rng.normal(loc=0.7, scale=0.1, size=30), "model7": rng.normal(loc=0.8, scale=0.1, size=30), "model8": rng.normal(loc=0.9, scale=0.1, size=30), } data = ( pd.DataFrame(dict_data) .rename_axis("cv_fold") .melt( var_name="estimator", value_name="score", ignore_index=False, ) .reset_index() ) ss.friedmanchisquare(*dict_data.values()) avg_rank = data.groupby("cv_fold").score.rank(pct=True).groupby(data.estimator).mean() test_results = sp.posthoc_conover_friedman( data, melted=True, block_col="cv_fold", group_col="estimator", y_col="score" ) # Significance heatmap for the example plt.figure() sp.sign_plot(test_results) plt.savefig("docs/source/_static/cd_diagram_example_sig_plot.png") # Third diagram plt.figure(figsize=(8, 2)) plt.title("Critical difference diagram of average score ranks") sp.critical_difference_diagram(avg_rank, test_results) plt.tight_layout() plt.savefig("docs/source/_static/cd_diagram1.png") # Fourth and final, stylized diagram plt.figure(figsize=(8, 2)) plt.title("Critical difference diagram of average score ranks") sp.critical_difference_diagram( ranks=avg_rank, sig_matrix=test_results, label_fmt_left="{label} [{rank:.3f}] ", label_fmt_right=" [{rank:.3f}] {label}", text_h_margin=0.3, label_props={"color": "black", "fontweight": "bold"}, crossbar_props={"color": None, "marker": "o"}, marker_props={"marker": "*", "s": 150, "color": "y", "edgecolor": "k"}, elbow_props={"color": "gray"}, ) plt.tight_layout() plt.savefig("docs/source/_static/cd_diagram2.png") ```I'll soon be adding some unit tests in another PR.I ended up adding the tests and some fixes to this same branch, I hope it doesn't get too cluttered to revise.
Here is a summary of the changes:
[DOC] A section is added to the tutorial briefly describing critical difference diagrams.
[FIX]
pyplot.annotate()
was replaced bypyplot.text()
in the plotter function. The former is sometimes not included in the figure bounding area when setting transparency or usingtight_layout()
, for example.[DOC, FIX] Improved function implementing Bron-Kerbosch algorithm to find crossbars for critical difference diagrams. Now simple input validation is performed and it does not enters infinite recursion when empty adjacency matrices are received.
[DOC] Docs for the aforementioned function and variable names also have improved readability.
[ENH] CD diagrams now plots the crossbars from the left-most position, keeping them more organized and consistent across runs.
[FIX] Previously, if a sample was different from all the others, a crossbar contained only it was still being created, and visible if a marker was set in crossbar_props.
critical_differece_diagram
now requires at least two samples to plot a crossbar.[MAINT] Some tests for CD diagrams were implemented.
Please let me know if there are any other modifications or additions that would be interesting.
Edit: spacing and new commits description.