search

ianfr / adsl-cpp

A Data Science Library for C++
BSD 2-Clause "Simplified" License
0 stars 0 forks source link

Dlib svm_c_trainer strange iris classifier results #8

Closed ianfr closed 3 years ago

ianfr commented 3 years ago

(Working on branch https://github.com/ianfr/adsl-cpp/tree/dlib-start) (Mentioned in commit https://github.com/ianfr/adsl-cpp/commit/2d991effcca3ce28f4b7a39b489c1d6483f81780)

All of the cross-entropy accuracies should not be 1, and the predictions should not be virtually identical.

BEGIN TRUNCATED PROGRAM OUTPUT: doing cross validation gamma: 1e-05 C: 1 cross validation accuracy: 1 1 gamma: 1e-05 C: 5 cross validation accuracy: 1 1 gamma: 1e-05 C: 25 cross validation accuracy: 1 1 gamma: 1e-05 C: 125 cross validation accuracy: 1 1 gamma: 1e-05 C: 625 cross validation accuracy: 1 1 gamma: 1e-05 C: 3125 cross validation accuracy: 1 1 gamma: 1e-05 C: 15625 cross validation accuracy: 1 1 gamma: 1e-05 C: 78125 cross validation accuracy: 1 1 gamma: 5e-05 C: 1 cross validation accuracy: 1 1 gamma: 5e-05 C: 5 cross validation accuracy: 1 1 gamma: 5e-05 C: 25 cross validation accuracy: 1 1 gamma: 5e-05 C: 125 cross validation accuracy: 1 1 gamma: 5e-05 C: 625 cross validation accuracy: 1 1 gamma: 5e-05 C: 3125 cross validation accuracy: 1 1 gamma: 5e-05 C: 15625 cross validation accuracy: 1 1 gamma: 5e-05 C: 78125 cross validation accuracy: 1 1 gamma: 0.00025 C: 1 cross validation accuracy: 1 1 gamma: 0.00025 C: 5 cross validation accuracy: 1 1 gamma: 0.00025 C: 25 cross validation accuracy: 1 1 gamma: 0.00025 C: 125 cross validation accuracy: 1 1 gamma: 0.00025 C: 625 cross validation accuracy: 1 1 gamma: 0.00025 C: 3125 cross validation accuracy: 1 1 gamma: 0.00025 C: 15625 cross validation accuracy: 1 1 gamma: 0.00025 C: 78125 cross validation accuracy: 1 1 gamma: 0.00125 C: 1 cross validation accuracy: 1 1 gamma: 0.00125 C: 5 cross validation accuracy: 1 1 gamma: 0.00125 C: 25 cross validation accuracy: 1 1 gamma: 0.00125 C: 125 cross validation accuracy: 1 1 gamma: 0.00125 C: 625 cross validation accuracy: 1 1 gamma: 0.00125 C: 3125 cross validation accuracy: 1 1 gamma: 0.00125 C: 15625 cross validation accuracy: 1 1 gamma: 0.00125 C: 78125 cross validation accuracy: 1 1 gamma: 0.00625 C: 1 cross validation accuracy: 1 1 gamma: 0.00625 C: 5 cross validation accuracy: 1 1 gamma: 0.00625 C: 25 cross validation accuracy: 1 1 gamma: 0.00625 C: 125 cross validation accuracy: 1 1 gamma: 0.00625 C: 625 cross validation accuracy: 1 1 gamma: 0.00625 C: 3125 cross validation accuracy: 1 1 gamma: 0.00625 C: 15625 cross validation accuracy: 1 1 gamma: 0.00625 C: 78125 cross validation accuracy: 1 1 gamma: 0.03125 C: 1 cross validation accuracy: 1 1 gamma: 0.03125 C: 5 cross validation accuracy: 1 1 gamma: 0.03125 C: 25 cross validation accuracy: 1 1 gamma: 0.03125 C: 125 cross validation accuracy: 1 1 gamma: 0.03125 C: 625 cross validation accuracy: 1 1 gamma: 0.03125 C: 3125 cross validation accuracy: 1 1 gamma: 0.03125 C: 15625 cross validation accuracy: 1 1 gamma: 0.03125 C: 78125 cross validation accuracy: 1 1 gamma: 0.15625 C: 1 cross validation accuracy: 1 1 gamma: 0.15625 C: 5 cross validation accuracy: 1 1 gamma: 0.15625 C: 25 cross validation accuracy: 1 1 gamma: 0.15625 C: 125 cross validation accuracy: 1 1 gamma: 0.15625 C: 625 cross validation accuracy: 1 1 gamma: 0.15625 C: 3125 cross validation accuracy: 1 1 gamma: 0.15625 C: 15625 cross validation accuracy: 1 1 gamma: 0.15625 C: 78125 cross validation accuracy: 1 1 gamma: 0.78125 C: 1 cross validation accuracy: 1 1 gamma: 0.78125 C: 5 cross validation accuracy: 1 1 gamma: 0.78125 C: 25 cross validation accuracy: 1 1 gamma: 0.78125 C: 125 cross validation accuracy: 1 1 gamma: 0.78125 C: 625 cross validation accuracy: 1 1 gamma: 0.78125 C: 3125 cross validation accuracy: 1 1 gamma: 0.78125 C: 15625 cross validation accuracy: 1 1 gamma: 0.78125 C: 78125 cross validation accuracy: 1 1

number of support vectors in our learned_function is 18 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: 1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Observed: -1 Predicted -0.0559632 Model accuracy: 50 END PROGRAM OUTPUT

ianfr commented 3 years ago

(see https://github.com/ianfr/adsl-cpp/commit/5080e47baa1546ddd5166f67562be0463f30ee9a)

It seems like the SVM is maybe overfitting the training data, resulting in the perfect cross-validation results.

Since cross-validation proved to be dangerous with this kind of model, binaryClassifier now uses a 50-25-25 training-testing-validation split, where the final accuracy is based on validation and model searching is done using testing accuracy.