search

heal-research / HeuristicLab

HeuristicLab - An environment for heuristic and evolutionary optimization
https://dev.heuristiclab.com
GNU General Public License v3.0
39 stars 16 forks source link

Show correlation of dataset features as HeatMap #1292

Closed HeuristicLab-Trac-Bot closed 11 years ago

HeuristicLab-Trac-Bot commented 13 years ago

Issue migrated from trac ticket # 1292

milestone: HeuristicLab 3.3.8 | component: Problems.DataAnalysis.Views | priority: medium | resolution: done

2010-11-22 15:57:45: @mkommend created the issue

HeuristicLab-Trac-Bot commented 13 years ago

2010-11-22 15:57:53: @mkommend changed status from new to accepted

HeuristicLab-Trac-Bot commented 13 years ago

2010-11-22 16:01:33: @gkronber commented

Please also add a view that shows the result of the Jarque-Bera test (normality test, implemented in alglib) for each variable.

HeuristicLab-Trac-Bot commented 13 years ago

2010-11-23 11:41:15: @mkommend changed status from accepted to assigned

HeuristicLab-Trac-Bot commented 13 years ago

2011-01-14 15:26:00: @s-wagner changed milestone from HeuristicLab 3.3.3 to HeuristicLab x.x.x

HeuristicLab-Trac-Bot commented 12 years ago

2012-06-05 09:23:02: @mkommend changed owner from mkommend to sforsten

HeuristicLab-Trac-Bot commented 12 years ago

2012-06-05 16:53:43: @gkronber commented

r7969: added HoeffdingsDependenceCalculator to calculate the non-parametric Hoeffding's dependency. Ideally it should be possible to show either Pearson's R², Spearman's rank correlation, or Hoeffding's dependency in the heat-map.

HeuristicLab-Trac-Bot commented 12 years ago

2012-06-18 16:04:38: @t-h-e commented

r8034: create branch to show correlation of dataset features[[BR]] r8035: branch project for implementing HeatMap to show correlation of dataset features[[BR]] r8036: branch another project for implementing HeatMap to show correlation of dataset features

r8038:

  • completed branch creation
  • first simple implementation of a HeatMap, which shows the correlation of the dataset features
HeuristicLab-Trac-Bot commented 12 years ago

2012-06-18 16:04:38: @t-h-e changed status from assigned to accepted

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-10 15:26:41: @t-h-e commented

r8276:

  • merged r8034 to r8179 from trunk
  • added BackgroundWorker
  • added ProgressBar
  • added SpearmansRankCorrelationCoefficientCalculator
  • corrected bug in HoeffdingsDependenceCalculator
  • made some changes in the GUI
HeuristicLab-Trac-Bot commented 12 years ago

2012-07-10 17:35:52: @gkronber commented

Please just use the alglib function for calculating the spearman's rank correlation Rename method 'Spear'

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-16 17:23:44: @t-h-e commented

r8294:

  • SpearmansRankCorrelationCoefficientCalculator now uses the alglib function
  • strings in ExtendedHeatMap have been made constant
HeuristicLab-Trac-Bot commented 12 years ago

2012-07-24 10:45:04: @t-h-e commented

r8318:

  • added cloning method and constructor to ExtendedHeatMap
  • renamed a variable in ExtendedHeatMapView
  • added backwards compatibility code in DataAnalysisProblemData
HeuristicLab-Trac-Bot commented 12 years ago

2012-07-24 10:47:28: @t-h-e changed owner from sforsten to mkommend

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-24 10:47:28: @t-h-e changed status from accepted to reviewing

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-24 10:58:47: @t-h-e changed milestone from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.8

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-25 10:52:35: @gkronber commented

  • Don't calculate the absolute value in Spearman's rank correlation.
  • Please add a property R or Correlation that simply returns the correlation coefficient in the Pearson's correlation calculator.
HeuristicLab-Trac-Bot commented 12 years ago

2012-07-27 16:13:00: @mkommend changed owner from mkommend to sforsten

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-27 16:13:00: @mkommend changed status from reviewing to assigned

HeuristicLab-Trac-Bot commented 12 years ago

2012-07-30 11:08:29: @gkronber commented

r8355:

  • fixed bugs in HoeffdingsDependenceCalculator
  • added test cases for HoeffdingsDependenceCalculator
HeuristicLab-Trac-Bot commented 12 years ago

2012-08-14 13:05:56: @t-h-e commented

r8483:

  • Renamed ExtendedHeatMap to FeatureCorrelation
  • deleted old CorrelationHeatMapView
  • added FeatureCorrelationView
HeuristicLab-Trac-Bot commented 12 years ago

2012-08-14 17:35:21: @t-h-e commented

r8492:

  • added TimeframeFeatureCorrelationView
HeuristicLab-Trac-Bot commented 12 years ago

2012-08-14 17:36:30: @t-h-e changed owner from sforsten to mkommend

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-14 17:36:30: @t-h-e changed status from assigned to reviewing

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-27 12:19:02: @mkommend commented

r8525: Added bin directory and resharper files to list of SVN excluded files.

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-27 14:09:18: @mkommend commented

r8526: Corrected build configurations in DatasetCorrelation branch.

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-27 18:00:07: @t-h-e commented

r8529:

  • BackgroundWorker is now reused in FeatureCorrelation
  • renamed some variables
  • ComboBoxes are now DropDownLists
  • FeatureCorrelation doesn't calculate the elements in the constructor anymore
  • small changes in the views
HeuristicLab-Trac-Bot commented 12 years ago

2012-08-29 16:44:32: @mkommend commented

r8537: Improved drawing of feature correlation view.

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-29 16:54:01: @mkommend commented

r8538: Merged trunk changes in preparation of the branch reintegration.

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-30 11:48:09: @mkommend commented

r8542: Integrated correlation analysis of datasets in the trunk.

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-30 11:53:01: @mkommend

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-30 11:53:01: @mkommend commented

The following things must be implemented:

  • Views of the same object is not synchronized
  • The default constructor doesn't assign a problem data to the feature correlation which could lead to exceptions
  • Use start and end values to calculate to correlation instead of strings declaring which partition should be used.
  • Remove the obsolete branch when all changes are implement.
HeuristicLab-Trac-Bot commented 12 years ago

2012-08-30 11:53:01: @mkommend changed owner from mkommend to sforsten

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-30 11:53:01: @mkommend changed status from reviewing to assigned

HeuristicLab-Trac-Bot commented 12 years ago

2012-08-30 11:56:29: @mkommend commented

r8543: Removed the feature correlation from the data analysis problem data as the implemenation is not yet finished and otherwise it could lead to persistence breaks.

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-03 16:16:21: @gkronber commented

r8559: removed the default constructor for FeatureCorrelation as it simply runs into a NullReferenceException (the default ctor is not used anywhere and is senseless).

This fixes the unit test fail for the meta-optimization branch on the builder.

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-05 16:07:25: @t-h-e commented

r8578:

  • added ProblemDataView which has a button to open the feature correlation
  • added abstract base class for feature correlations
  • added caches for the feature correlation
  • created own class for calculation of feature correlation
  • changed SelectedItemChanged to SelectionChangeCommitted events, so the correlation is only calculated if the user changes the selection
HeuristicLab-Trac-Bot commented 12 years ago

2012-09-05 16:11:36: @t-h-e commented

r8579 (not migrated): deleted obsolete branch

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-05 16:11:36: @t-h-e changed status from assigned to accepted

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-05 16:11:51: @t-h-e changed owner from sforsten to mkommend

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-05 16:11:51: @t-h-e changed status from accepted to reviewing

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-05 16:27:47: @t-h-e commented

r8581: removed unnecessary reference

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-16 23:45:19: @abeham commented

If possible, I suggest to limit the correlation analysis to only the allowed input variables plus the target variable. That way you can apply some filtering and it could help you iteratively refining your input variables.

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-18 15:29:38: @mkommend commented

The correlation analysis throws an exception if too few values were added to the used calculator.

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-18 15:30:23: @mkommend commented

Replying to [comment:33 abeham]:

If possible, I suggest to limit the correlation analysis to only the allowed input variables plus the target variable. That way you can apply some filtering and it could help you iteratively refining your input variables. This is a good point and should be implemented

HeuristicLab-Trac-Bot commented 12 years ago

2012-09-24 18:48:31: @t-h-e commented

r8689:

  • NaN values are used, if the calculation is invalid (e.g. missing values, infinity etc.)
  • Variables can now be filtered. Initially allowed input variables and target variable are shown, but with a right click a dialog can be opened to select variables, which shall be shown
HeuristicLab-Trac-Bot commented 12 years ago

2012-09-25 11:40:26: @abeham commented

I have a few remarks:

  • I would restrict Pearsons R2 to only use green-yellow-red colors. It's a bit confusing that in Pearsons R green means no correlation, but in R2 it means medium correlation while red still retains its meaning.
  • Hoeffdings Dependence doesn't have 1s in the diagonal (why?)
  • Numbers are not easily readable if they're on dark-blue background
HeuristicLab-Trac-Bot commented 12 years ago

2012-09-25 11:42:27: @gkronber commented

Replying to [comment:37 abeham]:

I have a few remarks:

  • Hoeffdings Dependence doesn't have 1s in the diagonal (why?)

This is correct behaviour when the variable contains duplicate values.

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 11:28:43: @mkommend commented

r8728: Corrected SpearmansRankCalculator.

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 11:43:41: @mkommend commented

r8729: Moved FeatureCorrelation specific classes from Problems.DataAnalysis to Problems.DataAnalysis.Views.

HeuristicLab-Trac-Bot commented 12 years ago

2012-10-04 11:50:08: @mkommend commented

Currently the the TimeFrameCorrelationView is displayed by default instead of the "normal" CorrelationView. Furthermore we should discuss the source code in detail.