search

heal-research / HeuristicLab

HeuristicLab - An environment for heuristic and evolutionary optimization
https://dev.heuristiclab.com
GNU General Public License v3.0
39 stars 16 forks source link

Model Comparison for Classification #1998

Closed HeuristicLab-Trac-Bot closed 9 years ago

HeuristicLab-Trac-Bot commented 11 years ago

Issue migrated from trac ticket # 1998

milestone: HeuristicLab 3.3.13 | component: Algorithms.DataAnalysis | priority: high | resolution: done

2012-12-17 17:09:17: @t-h-e created the issue

Similar to Regression, it shall be possible to easily create simple models for classification, which a classification model has to exceed to be accepted as good model.

0R, 1R and LDA classifiers will be created to compare them to the hopefully better classification model.

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-17 17:32:30: @t-h-e changed status from new to accepted

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-17 17:32:30: @t-h-e commented

r9069: initial commit of new branch

r9070: branch project HeuristicLab.Algorithms.DataAnalysis

r9071: finished preparing branch

HeuristicLab-Trac-Bot commented 11 years ago

2012-12-18 16:15:48: @t-h-e commented

r9073: branch project Problems.DataAnalysis

r9074:

  • added ZeroR and OneR classifiers
  • added ConstantClassificationModel/-Solution and OneRClassificationModel/-Solution

A view to display the OneRClassificationModel has to be added.

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-07 17:34:09: @t-h-e commented

r9116: branch project Problems.DataAnalysis.Views

r9117: branch project Algorithms.DataAnalysis.Views

r9119:

  • added OneRClassificationModelView
  • added ClassificationSolutionComparisonView
  • added several calculators (ConfusionMatrixCalculator, FOneScoreCalculator, MatthewsCorrelationCoefficientCalculator)
  • fixed bug in OneR
  • added StorableClass and Item attribute to several classes
HeuristicLab-Trac-Bot commented 11 years ago

2013-01-09 16:27:27: @t-h-e changed status from accepted to reviewing

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-09 16:27:27: @t-h-e changed owner from @t-h-e to @mkommend

HeuristicLab-Trac-Bot commented 11 years ago

2013-01-09 16:27:27: @t-h-e commented

r9135:

  • OneR now handles missing values separately
  • adapted OneRClassificationModelView to show the class of missing values
  • with a double-click on the row header in ClassificationSolutionComparisonView the selected solution opens in a new view
  • put a try catch block around linear discriminant analysis solution (it is only shown, if it doesn't throw an exception)

ZeroR may not be a good model in the ClassificationSolutionComparisonView, because MatthewsCorrelationCoefficientCalculator will always calculate NaN and F1 score will also be NaN if the negative class is chosen by ZeroR.

HeuristicLab-Trac-Bot commented 11 years ago

2013-08-07 13:59:34: @gkronber changed owner from @mkommend to @gkronber

HeuristicLab-Trac-Bot commented 11 years ago

2013-10-08 17:54:34: @gkronber changed status from reviewing to assigned

HeuristicLab-Trac-Bot commented 11 years ago

2013-10-08 17:54:34: @gkronber changed owner from @gkronber to @s-wagner

HeuristicLab-Trac-Bot commented 11 years ago

2013-10-08 17:54:34: @gkronber commented

Can we queue this for the next release version?

HeuristicLab-Trac-Bot commented 10 years ago

2013-12-18 17:37:08: @gkronber changed owner from @s-wagner to @mkommend

HeuristicLab-Trac-Bot commented 10 years ago

2013-12-18 17:37:08: @gkronber commented

Discussed in architects meeting, should be integrated into the trunk.

HeuristicLab-Trac-Bot commented 10 years ago

2014-02-26 15:37:17: @mkommend changed milestone from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.10

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-05 17:31:28: @mkommend commented

r10553: Updated classification model comparison branch with trunk changes.

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-05 17:36:31: @mkommend commented

r10556: Updated classification model comparison branch with trunk changes (remaining changes).

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-06 11:12:35: @mkommend commented

r10560: Fixed bugs in classification solution comparison view.

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-10 14:44:05: @mkommend commented

r10568: Code cleanup in ZeroR classification algorithm.

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-11 15:17:20: @mkommend commented

r10569: Reimplemented OneR classification algoithm.

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-11 15:41:51: @mkommend commented

r10570: Added missing value handling in new implementation of OneR.

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-11 15:52:39: @mkommend changed status from assigned to reviewing

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-11 15:52:39: @mkommend changed owner from @mkommend to @gkronber

HeuristicLab-Trac-Bot commented 10 years ago

2014-03-11 15:52:39: @mkommend commented

Please review r10560, r10568 to r10570. All other changesets have already been reviewed.

For comparison reasons I left the original version of the OneR algorithm untouched.

HeuristicLab-Trac-Bot commented 10 years ago

2014-05-21 17:11:48: @gkronber changed status from reviewing to assigned

HeuristicLab-Trac-Bot commented 10 years ago

2014-05-21 17:11:55: @gkronber changed status from assigned to accepted

HeuristicLab-Trac-Bot commented 10 years ago

2014-05-21 17:21:35: @gkronber commented

Reviewed r10560 changes are OK:

Made a more detailed review of the source code. Notes from the overall source code review:

  • Adapt to work with current trunk version (export button).
  • Exception occurs when running a linear discriminant analysis model (works for the trunk) cannot reproduce
  • Move OneR and ZeroR out of the Algorithms.DataAnalysis.Linear folder. done
  • OneR and ZeroR should not be available in the "New Item" dialog. done
  • ExecutionTime for OneR should not be produced as a result. I guess this is ok.
  • The file OneRTest.cs can be deleted? No, the OneRTest.cs file is the new implementation by mkommend. The file OneR.cs should be deleted
HeuristicLab-Trac-Bot commented 10 years ago

2014-05-21 17:21:35: @gkronber

HeuristicLab-Trac-Bot commented 10 years ago

2014-05-21 17:21:35: @gkronber

HeuristicLab-Trac-Bot commented 10 years ago

2014-05-21 17:21:35: @gkronber

HeuristicLab-Trac-Bot commented 10 years ago

2014-07-06 12:01:26: @gkronber changed milestone from HeuristicLab 3.3.10 to HeuristicLab 3.3.11

HeuristicLab-Trac-Bot commented 10 years ago

2014-07-06 12:01:26: @gkronber commented

Not yet ready for release

HeuristicLab-Trac-Bot commented 9 years ago

2015-01-21 19:09:38: @gkronber changed milestone from HeuristicLab 3.3.11 to HeuristicLab 3.3.x Backlog

HeuristicLab-Trac-Bot commented 9 years ago

2015-03-09 10:37:22: @mkommend changed milestone from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.12

HeuristicLab-Trac-Bot commented 9 years ago

2015-03-09 14:35:31: @mkommend changed status from accepted to readytorelease

HeuristicLab-Trac-Bot commented 9 years ago

2015-03-09 14:35:31: @mkommend changed owner from @gkronber to @mkommend

HeuristicLab-Trac-Bot commented 9 years ago

2015-03-09 14:35:50: @mkommend changed status from readytorelease to assigned

HeuristicLab-Trac-Bot commented 9 years ago

2015-03-09 14:35:50: @mkommend changed owner from @mkommend to @gkronber

HeuristicLab-Trac-Bot commented 9 years ago

2015-05-04 13:20:58: @gkronber changed priority from medium to high

HeuristicLab-Trac-Bot commented 9 years ago

2015-05-04 13:20:58: @gkronber changed status from assigned to accepted

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-07 13:10:44: @gkronber changed milestone from HeuristicLab 3.3.12 to HeuristicLab 3.3.13

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-07 13:10:44: @gkronber commented

Sorry

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 18:23:34: @gkronber commented

r13081: changed framework version to v4.5

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 18:29:23: @gkronber commented

r13082: merged changesets r10553 to r13081 (only on HeuristicLab.Problems.DataAnalysis.Views) from trunk to branch r13083: merged changesets r10551 to r13082 (only on HeuristicLab.Problems.DataAnalysis) from trunk to branch r13084: merged changesets r10551 to r13083 (only on HeuristicLab.Algorithms.DataAnalysis.Views) from trunk to branch r13085: merged changesets r10551 to r13084 (only on HeuristicLab.Algorithms.DataAnalysis) from trunk to branch

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 18:29:23: @gkronber

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 18:29:23: @gkronber

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 18:29:23: @gkronber

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 18:45:22: @gkronber commented

r13086: made compatibility changes necessary because of trunk developments (compile fail)

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 19:48:36: @gkronber commented

r13089

  • deleted obsolete version of OneR algorithm (also does perform worse than mkommend's implementation in my tests)
  • reused the ConstantRegressionModel as ConstantClassificationModel (OK?)
  • fixed a few strings here and there
HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 20:26:39: @gkronber commented

r13090: moved OneR and ZeroR out of the folder for linear models, renamed OneRTest -> OneR

HeuristicLab-Trac-Bot commented 9 years ago

2015-10-29 20:39:26: @gkronber commented

r13091: minor changes while reviewing

Reviewed changes in

  • HeuristicLab.Problems.DataAnalysis (online calculators of F1-score and Matthews correlation have been added)
  • HeuristicLab.Algorithms.DataAnalysis (ZeroR and OneR algorithms have been added)
  • HeuristicLab.Problems.DataAnalysis.Views (only ClassificationSolutionComparisonView has been added)
  • HeuristicLab.Algorithms.DataAnalysis.Views (only the OneRClassificationModelView has been added)

TODO:

  • don't calculate F1 score for multi-class problems
  • terminate branch
  • more detailed review of online calculators and algorithms
  • add F1 score and Matthew's correlation to classification results collection
  • mark ConstantRegressionModel obsolete and create new class: ConstantModel