search

nfmcclure / tensorflow_cookbook

Code for Tensorflow Machine Learning Cookbook
https://www.packtpub.com/big-data-and-business-intelligence/tensorflow-machine-learning-cookbook-second-edition
MIT License
6.23k stars 2.41k forks source link

Kernel SVM training with data in a fixed sequence #158

Closed Gourmentic closed 2 years ago

Gourmentic commented 5 years ago

Hello, I find a very interesting thing: after changing the code of kernel SVM as below, a better training progress and a better performance is shown.

rand_index = np.random.choice(len(x_vals), size=batch_size)

rand_x = x_vals[rand_index]

rand_y = np.transpose([y_vals[rand_index]])

rand_x = x_vals rand_y = y_vals.reshape(-1, 1)

image image image image

I think the reason is that every scalar bi in Vector Variable b should follow the ith data pair(feature,label) according to the dual formula. But random choice makes the weight bi be trained by different data pair ,not only the ith and maybe give a wrong direction to update bi. image

And tatnguyennguyen give the reason why the current code get a good result. https://github.com/nfmcclure/tensorflow_cookbook/issues/104

So the current code is only ok for the whole-batch training. Maybe embedding layer would be used for a mini batch training. :)

Gourmentic commented 2 years ago

SGD can not be used as optimizer of Kernel SVM directly, the mentioned code in the book should be changed.