Evaluator for classification problems that considers classification penalties

[HeuristicLab-Trac-Bot]

Issue migrated from trac ticket # 1924

milestone: HeuristicLab 3.3.8 | component: Problems.DataAnalysis.Symbolic.Classification | priority: medium | resolution: done

2012-08-30 12:12:55: @abeham created the issue

---

According to mkommend, the penalties are currently only accounted for in the thresholds. An evaluator should be created that calculates the penalty score which is to be minimized by the algorithm.

[HeuristicLab-Trac-Bot]

2012-08-30 12:13:36: @abeham changed owner from @mkommend to @abeham

[HeuristicLab-Trac-Bot]

2012-08-30 12:13:36: @abeham changed status from new to accepted

[HeuristicLab-Trac-Bot]

2012-08-31 12:14:00: @abeham commented

---

r8548: Added evaluator

[HeuristicLab-Trac-Bot]

2012-08-31 12:14:00: @abeham changed owner from @abeham to @mkommend

[HeuristicLab-Trac-Bot]

2012-08-31 12:14:00: @abeham changed status from accepted to reviewing

[HeuristicLab-Trac-Bot]

2012-08-31 13:06:58: @abeham commented

---

divide score by evaluated samples

[HeuristicLab-Trac-Bot]

2012-08-31 13:06:58: @abeham changed owner from @mkommend to @abeham

[HeuristicLab-Trac-Bot]

2012-08-31 13:06:58: @abeham changed status from reviewing to assigned

[HeuristicLab-Trac-Bot]

2012-08-31 13:47:50: @abeham commented

---

r8549: Divided penalty by the number of evaluated fitness cases

[HeuristicLab-Trac-Bot]

2012-08-31 13:47:50: @abeham changed owner from @abeham to @mkommend

[HeuristicLab-Trac-Bot]

2012-08-31 13:47:50: @abeham changed status from assigned to reviewing

[HeuristicLab-Trac-Bot]

2012-08-31 14:03:22: @mkommend commented

---

r8551: Adapted evaluator to changes in r8550 (#1919).

[HeuristicLab-Trac-Bot]

2012-08-31 14:03:22: @gkronber

[HeuristicLab-Trac-Bot]

2012-08-31 14:03:22: @gkronber

[HeuristicLab-Trac-Bot]

2012-09-03 13:42:40: @mkommend edited the issue description

[HeuristicLab-Trac-Bot]

2012-09-12 16:20:57: @mkommend commented

---

r8636:

• Changed the accuracy threshold calculator to eliminate the necessity that the class values are ordered.
• Adapted the symbolic classification simplifier to work with all ISymbolicClassificationModels.
• Corrected ROCCurvesView to also work if the class values are not sorted.

[HeuristicLab-Trac-Bot]

2012-09-12 16:20:57: @gkronber

[HeuristicLab-Trac-Bot]

2012-09-12 16:21:27: @mkommend commented

---

Please test the changes and forward the ticket to gkronber for review.

[HeuristicLab-Trac-Bot]

2012-09-12 16:21:27: @mkommend changed owner from @mkommend to @abeham

[HeuristicLab-Trac-Bot]

2012-09-12 17:08:14: @mkommend commented

---

r8637: Added comment that explains the usage of the cloner in the classification simplifier view.

[HeuristicLab-Trac-Bot]

2012-09-13 17:38:31: @mkommend commented

---

r8646: Corrected Pearson's R² evaluator for classification (estimated values were not limited in quality calculation)

[HeuristicLab-Trac-Bot]

2012-09-13 17:45:54: @abeham commented

---

r8636 to r8637 would better fit in #1928 in my opinion, r8646 also seems somewhat of a rogue change here.

Anyway, I got an exception when opening the ROC Curves view, I assume as a consquence of r8636:

System.ArgumentException: A chart element with the name 'Class 1' already exists in the 'SeriesCollection'.
   at System.Windows.Forms.DataVisualization.Charting.ChartNamedElementCollection`1.InsertItem(Int32 index, T item)
   at HeuristicLab.Problems.DataAnalysis.Views.DiscriminantFunctionClassificationRocCurvesView.UpdateChart() in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Classification\DiscriminantFunctionClassificationRocCurvesView.cs:line 158
   at HeuristicLab.Problems.DataAnalysis.Views.DiscriminantFunctionClassificationRocCurvesView.OnContentChanged() in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Classification\DiscriminantFunctionClassificationRocCurvesView.cs:line 89
   at HeuristicLab.MainForm.WindowsForms.ContentView.set_Content(IContent value) in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Views\ContentView.cs:line 51
   at HeuristicLab.MainForm.WindowsForms.ViewHost.OnContentChanged() in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Controls\ViewHost.cs:line 146
   at HeuristicLab.MainForm.WindowsForms.ContentView.set_Content(IContent value) in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Views\ContentView.cs:line 51
   at HeuristicLab.Problems.DataAnalysis.Views.DataAnalysisSolutionView.itemsListView_SelectedIndexChanged(Object sender, EventArgs e) in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Solution Views\DataAnalysisSolutionView.cs:line 116
   at System.Windows.Forms.ListView.WmReflectNotify(Message& m)
   at System.Windows.Forms.ListView.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[HeuristicLab-Trac-Bot]

2012-11-06 18:30:56: @abeham commented

---

I did test the changes and didn't find any more issues.

[HeuristicLab-Trac-Bot]

2012-11-06 18:30:56: @abeham changed owner from @abeham to @gkronber

[HeuristicLab-Trac-Bot]

2012-11-28 15:10:52: @gkronber changed status from reviewing to readytorelease

[HeuristicLab-Trac-Bot]

2012-11-28 15:10:52: @gkronber commented

---

Reviewed r8646, the whole evaluation procedure for regression and classification models has been unified in ticket #1951.

Reviewed r8636 to r8637 which indeed does not fit into this ticket.

Reviewed r8548 to r8549 and r8551. The changes look good.

In the future, please make sure that correct revision numbers are stated in the ticket comments! I had to change several comments and search correct revision numbers from the time line.

[HeuristicLab-Trac-Bot]

2013-05-10 01:54:22: @s-wagner changed status from readytorelease to closed

[HeuristicLab-Trac-Bot]

2013-05-10 01:54:22: @s-wagner removed resolution