pavel-krivanek
commented
5 years ago 
Just wanted to share my favourite plot lately: ECDF with confidence band.
I see a family of plots here:
- Density plot: Show best guess distribution, but has no confidence information.
- Violin plot: Show best guess distribution, plus parametric confidence for comparing averages.
- Banded (E)CDF: Show the set of likely distributions, with no parametric assumptions.
For my work in comparing software performance across a range of benchmarks I have found the last option by far the best. Density plot breaks down because I need the confidence interval (i.e. do I have enough data to draw an inference) and Violin plot doesn't fit because the data is not homogenous(?) enough to be summarized by statistics like mean/median/quartiles.
Some more background at https://stats.stackexchange.com/questions/298290/plotting-non-parametric-ecdef-confidence-envelopes-for-comparison. The confidence bands are actually surprisingly easy to compute.
Sorry to distract from the awesome work you guys are doing! :-)
akevalion
https://datavizcatalogue.com/methods/violin_plot.html