heal-research / HeuristicLab

HeuristicLab - An environment for heuristic and evolutionary optimization
https://dev.heuristiclab.com
GNU General Public License v3.0
32 stars 15 forks source link

Implement views for statistical hypothesis testing #2031

Closed HeuristicLab-Trac-Bot closed 9 years ago

HeuristicLab-Trac-Bot commented 11 years ago

Issue migrated from trac ticket # 2031

milestone: HeuristicLab 3.3.11 | component: Analysis | priority: medium | resolution: done

2013-04-10 12:28:55: @Shabbafru created the issue


Besides using the charts of the RunCollectionView for analyzing runs there should also be the option to do statistical significance testing. Therefore new RunCollection views should be implemented that offer this functionality.

~~TODO: Check http://powerandsamplesize.com for a better way to do sample size estimation. ~~

HeuristicLab-Trac-Bot commented 9 years ago

2014-07-18 19:55:29: @abeham commented


I just took a quick look at this. Generally, I'd say this is quite a good addition. Very nice workflow.

  • The yellow exclamation mark icon should have a tooltip that explains it. I see "Data is normally distributed: !". It seems that the exclamation marks says this is not the case, but at what significance level?
  • p-values should be rounded to 4 places after the point. Otherwise the text box might be too small to fit the whole number: Often the important part (exponent) is at the end. Instead of 2.93559215344167E-05 which would be chopped off to e.g. "2.9355921534416..." it should say 2.9356E-05. The text boxes should be at least as wide to allow these 10 characters to fit nicely.
  • The histogram didn't look to be so useful, maybe a stacked histogram would be more useful with common bins for all
  • The p-value adjustment needs more explanation. It seems that you have used [correction] for the post-hoc analysis. In the Mann-Whitney-U test most got adjusted from 0.0001 to 0.0007 as you would expect comparing to 7 other groups, but one adjusted p-value remained the same (0.00423...) and another one was adjusted from 0.00025.. to 0.0007. Above all, it seems strange to obtain such p-Values, it rather seems as if this was the significance level.
  • I don't see a possibility to set the significance level.
HeuristicLab-Trac-Bot commented 9 years ago

2014-07-18 19:55:29: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-07-18 19:55:29: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-07-18 19:55:29: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-15 17:13:35: @Shabbafru changed status from reviewing to assigned

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-15 17:13:35: @Shabbafru changed owner from @abeham to @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-15 17:20:46: @Shabbafru commented


r11368 adapted sample size view to the changes of #2120

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-17 12:53:02: @Shabbafru commented


r11375 updated license headers and version

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-17 13:34:22: @Shabbafru commented


r11376 fixed RunCollection event handling in ChartAnalysisView

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-17 16:24:46: @Shabbafru edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-17 17:35:00: @Shabbafru commented


r11378 improved correlation view and some clean ups

HeuristicLab-Trac-Bot commented 9 years ago

2014-09-17 20:55:13: @Shabbafru commented


r11379 fixed some bugs

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru commented


mkommends review comments:

  • Statistical Testing View:

    • Check if icons are correct. The label should also change according to the icon and should be also displayed for the tests and not only for normal distribution test
    • Add links in help view that describe the used algorithms in detail
    • Adjusted p-values are sometimes equal to p-values. Check if this is correct
    • t-Test values should be gray when used with data that is not normally distributed
    • For the normal distribution check, a check mark should only be given if all are normally distributed
    • Inherit StringConvertibleMatrixView and fix the column with or adjust to length of values
    • If an error occurs and the "not enough samples" dialog is shown, the loading bar is not hidden afterwards
    • If possible, a kernel density curve should also be displayed in the histogram
    • Check t-test for one/two sided
  • Correlations View:

    • Should display the values as matrix, the method should be chooseable
    • Sort inside results and parameters
  • Chart Analysis View:

    • Percentiles seem to be switched
    • Change Relativ Error to Average Relative Error
    • log values should be removed from data table
    • avg/upper/lower values should be descriped in the help text
  • Sample Size Influence View

    • add help text
    • used draw without putting back from StatisticsEnumerable
  • On trunk integration, move algorithms to Analysis plugin, create an own plugin for the views

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-06 12:46:44: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-11-28 16:14:31: @Shabbafru commented


r11601

  • fixed column width of p-values
  • started working on drawing a normal distribution over the histogram
HeuristicLab-Trac-Bot commented 9 years ago

2014-12-01 22:10:13: @Shabbafru commented


r11611 adapted statistical testing view to new histogram

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-01 22:47:31: @Shabbafru commented


r11612 fixed a bug in the progress handling of the statistical testing view

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-02 21:31:21: @Shabbafru commented


r11625 switched statistical testing branch to .NET 4.5

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-04 10:59:57: @Shabbafru commented


r11644 implemented review comments for correlations view

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-05 15:54:46: @Shabbafru commented


r11665 implemented review comments for chart analysis view

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-08 23:31:51: @Shabbafru commented


r11670 worked on sample size influence view

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-09 00:49:38: @Shabbafru commented


r11671 added documentation for sample size influence view

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-09 15:16:50: @Shabbafru commented


r11673 fixed a small bug in Bonferroni-Holm adjustment and added more unit tests for it

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-16 23:55:58: @Shabbafru commented


r11691

  • expanded documentation for statistical testing view
  • changed t-test to unpooled method
  • removed sample size determination for t-test as this is probably not correct
HeuristicLab-Trac-Bot commented 9 years ago

2014-12-17 11:31:57: @Shabbafru commented


r11692

  • fixed a bug in Cohens d / Hedges g calculation
  • fixed calculation of pairwise tests (no more columns with only zeroes)
  • some refactoring
HeuristicLab-Trac-Bot commented 9 years ago

2014-12-17 13:36:24: @Shabbafru commented


r11693 added dialog for configuring the SignificanceLevel and renamed view

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-18 11:53:32: @Shabbafru commented


r11695 added more information to the UI about the results of the statistical tests

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-18 12:48:57: @Shabbafru commented


Replying to [comment:54 ascheibe]:

I did not do graying out the values of the t-test if the data is not normally distributed. It seems that if you have bigger sample sizes it is not such a problem. See http://stats.stackexchange.com/questions/9573/t-test-for-non-normal-when-n50

mkommends review comments: ...

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-18 12:48:57: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-18 16:12:52: @Shabbafru commented


r11696

  • improved code of statistical testing view
  • improved documentation
HeuristicLab-Trac-Bot commented 9 years ago

2014-12-18 16:56:17: @Shabbafru commented


r11697 improved code of views

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 12:46:15: @Shabbafru commented


r11699

  • fixed confidence intervals calculation
  • added more unit tests
  • some cosmetic changes
HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 13:29:26: @Shabbafru commented


r11702 added unit tests for effect size measures

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 13:49:09: @Shabbafru commented


Trunk integrationr11703 moved statistic algs and unit tests to trunk

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 13:49:09: @Shabbafru

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 13:56:38: @Shabbafru commented


r11704 moved annotations-default icon to common.resources

HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 14:42:14: @Shabbafru commented


r11705

  • moved statistics views to a new plugin (HL.Analysis.Statistics.Views) in trunk
  • fixed namespaces of unit tests
HeuristicLab-Trac-Bot commented 9 years ago

2014-12-19 15:28:04: @Shabbafru commented


r11706 (not migrated) deleted statistical testing branch