Deeper understanding of Weka - Error Metrics

[damirsaleschoice]

Open a non-nominal example train with any algorithm

You will see an output such as Correlation coefficient 0.937 Mean absolute error 30846523146.2126 Root mean squared error 38225020343.6509 Relative absolute error 32.4817 % Root relative squared error 34.8704 % Total Number of Instances 144

Learn what each one of them means, and what role they play in the error metrics.

• What in your opinion would be the best error metrics ?
• What in your opinion would be the error metrics displaying an over-trained model ?

[sherlockliang888]

It depends on how we want to treat outliers I guess: if we want to ignore them then MAE is a better choice, but if we want to account for them then MSE/RMSE may be a better choice. I would prefer to use RMSE as it shows differences in both directions, and it is more commonly used (I think). I know that we can learn whether a model is over-trained or not from validation set, but I'm not too sure which metrics to look at for it...but I think when the number of instances is small, the model might be over-trained.

[damirsaleschoice]

Cool. I usually just look at the Correlation coefficient, and if it is too high (beyond 0.95) then there is probably some over-training involved... If too low, then it means that the model is not so good... only then I look at the other metrics.

I will close this ticket as DONE

[sherlockliang888]

This is helpful, thanks.