search

jeffheaton / encog-java-core

http://www.heatonresearch.com/encog
Other
742 stars 268 forks source link

SOM Training: suspiciously low validation error #82

Closed PetrToman closed 12 years ago

PetrToman commented 12 years ago
  1. In Workbench, create .ega file for SOM classification - using data: http://dione.zcu.cz/~toman40/encog/data7.zip - and set "Maximum Error Percent (0-100)" = 10.
  2. Excecute "task-full".
  3. Create data7_eval.egb by "Normalize to Training(EGB)" action.
  4. Double click data7_train.eg, click "Train", set "Training Set" = data7_train.egb and "Validation Set (optional)" = data7_eva.egb and select "Basic... Training".
  5. Run the training again -> "Validation Error" will be ~7.5 %, but there is only ~50 % of correctly classified rows in data7_output.csv (where y=Output:y).
jeffheaton commented 12 years ago

This is actually correct behavior for a SOM. SOM's are very different than most of the ML methods in Encog. I probably should document this better, because the workbench allows you to simply drop a SOM in place just like a SVM or feedforward.

SOM's are unsupervied. So, the ideal column is ignored. What a SOM tries to do is group the data into a number of clusters equal to the output neuron count. Unfortunately, what output neuron gets mapped to which cluster is very hit-or-miss, depending on how the random weights were set. So comparing to expected outputs is difficult because the class number can change. Also the SOM ignores ideal, so it really has no notion of what group the data should have joined. In this way SOM acts like a clustering algorithm, similar to kMeans. Except, unlike kMeans, SOM can tell you which group it feels a new piece of data most closely resembles.

Also, the error is calculated very differently. The ideal data is not used. The method used is briefly described here. http://www.heatonresearch.com/wiki/SOM_Error

PetrToman commented 12 years ago

I see, thank you for your explanation. I haven't played much with SOMs, so I thought it might be some enhancement (like SVMs were originally capable only of classification). Btw, there is a typo on the wiki page - "ere is no standard".