search

ashokpant / dkpro-tc

Automatically exported from code.google.com/p/dkpro-tc
Other
0 stars 0 forks source link

Create a dedicated evaluation object/module #50

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Some reports might produce mandatory output which other tasks or reports 
(potentially) depend on. This is the case for some of the reports which create 
the results of the test task (f-scores etc.). We should think about moving this 
(core) functionality to their an own task.

Original issue reported on code.google.com by daxenber...@gmail.com on 17 Sep 2013 at 4:41

GoogleCodeExporter commented 9 years ago 
Agreed.  See comments on r704.
I am particularly interested in moving the calculations out of 
BatchCrossValidationReport, and into an Object that all Reports can access.

Original comment by EmilyKJa...@gmail.com on 21 Mar 2014 at 6:25

GoogleCodeExporter commented 9 years ago 
A task may be a bit heavy, but creating some separate classes for data 
structures commonly used in reports and for evaluations over these data 
structures appears sensible. The structures and functionality could easily be 
reused and the reports would become more light-weight.

Original comment by richard.eckart on 23 Mar 2014 at 12:44

GoogleCodeExporter commented 9 years ago 
We need to create a central Evaluation Object in TC, which will serve as a 
connector between the machine learning framework and the evaluation (i.e. all 
reports). 

Why?
1) This Evaluation Object can be used to create a file in a format which can be 
imported into an external program to do significance tests (issue 112).
2) To avoid bias in the aggregation of results from CV folds, an overall 
confusion matrix should be created which is used to further calculate F1 etc. 
An Evaluation Object can also hold the overall confusion matrix (issue 113).

Original comment by daxenber...@gmail.com on 22 Apr 2014 at 3:42

GoogleCodeExporter commented 9 years ago

Original comment by daxenber...@gmail.com on 22 Apr 2014 at 3:48

GoogleCodeExporter commented 9 years ago

Original comment by daxenber...@gmail.com on 22 Apr 2014 at 3:49

GoogleCodeExporter commented 9 years ago

Original comment by daxenber...@gmail.com on 4 Jun 2014 at 12:34

GoogleCodeExporter commented 9 years ago

Original comment by daxenber...@gmail.com on 5 Sep 2014 at 8:45

GoogleCodeExporter commented 9 years ago 
This issue was updated by r1133 and r1134.

Original comment by daxenber...@gmail.com on 8 Oct 2014 at 9:12

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1136.

tests for soft/strict evaluation.

Original comment by daxenber...@gmail.com on 8 Oct 2014 at 9:48

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1137.

adding TODOs.

Original comment by daxenber...@gmail.com on 8 Oct 2014 at 10:00

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1289.

introducing a generic multi-label result wrapper to work with the latest 
version of meka and DKPro TC's new evaluation module

Original comment by daxenber...@gmail.com on 10 Dec 2014 at 8:54

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1359.

Created a test for the new evaluation report, in part. outcome id report

Original comment by daxenber...@gmail.com on 17 Mar 2015 at 11:47

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1366.

Expanding functionality of the evaluation module/helper classes.
Work in progress.

Original comment by daxenber...@gmail.com on 17 Mar 2015 at 4:16

GoogleCodeExporter commented 9 years ago

Original comment by daxenber...@gmail.com on 27 Mar 2015 at 1:05

GoogleCodeExporter commented 9 years ago 
Good reference for multi-label evaluation and calculation of scores: 
http://scikit-learn.org/stable/modules/model_evaluation.html#classification-metr
ics

Original comment by daxenber...@gmail.com on 30 Mar 2015 at 8:24

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1396.

restructuring evaluation module; mostly multi-label part
several tests are broken/ignored atm, needs to be investigated

Original comment by daxenber...@gmail.com on 2 Apr 2015 at 4:04

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1400.

enabling crossvalidation setup with evaluation module
adding test for crossvalidation setup with evaluation module

Original comment by daxenber...@gmail.com on 3 Apr 2015 at 3:09

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision r1404.

copy all relevant discriminators into CV report on batch task level;
javadoc

Original comment by daxenber...@gmail.com on 7 Apr 2015 at 2:05