Closed giuseta closed 8 years ago
Recap of what we just discussed: Seems it makes more sense to put the validation result type into a learning system configuration as for example learningsystems/dllearner/system.ini such that the benchmark framework knows a priori what kind of validation result to expect.
Not all the classifiers are discrete, some of them yield an instance score or probability. In these cases the ROC and PR curves (and their areas) are usually used. Therefore I suggest that the configuration file produced by
./validate
script should have a field that states the type of result (binary or score) and, in case it is a score type, a list of couples(<type_example>,<score>)
should be given, where<type_example>
could be+
or-
and<score>
is obviously the numeric value. For example:Any suggestions/comments?