Closed jurcicek closed 11 years ago
This is gong to be a great example to add to the code. I think it should work with the latest code in master though -- could you try this again against master and update this PR without the revert if that works ?
Well, the pull request is rather messy. However, I do not know how to make it better. In short, it should just add xor_classifier.py example.
I cleaned up the pull request.
Today I verified that the neural network is indeed capable of computing the XOR function. In your example script, I added the following lines between creating the Experiment and calling .run()
:
e.network.weights[0].set_value(np.array([[100, 100], [100, 100]]))
e.network.biases[0].set_value(np.array([-70, -130]))
e.network.weights[1].set_value(np.array([[1], [-1]]))
e.network.biases[1].set_value(np.array([0]))
(Also, make sure that you are using layers=(2, 2, 1)
.)
This sets the parameters of the network to fixed values that will compute the XOR function, and it does actually work.
The next question to address is, why is the optimization procedure not finding these (or equivalent) weights ? It's a very difficult optimization problem, since there are (in this network) 11 unknowns, but only 4 equations. At any rate, I'll have a look at it in the next few days.
Hi,
I have an example of XOR NN in pybrain. Please see the attachment. xor_pybrain_classifier.pyhttps://docs.google.com/file/d/0Bx01_P9NXYisRGxXUi1TaVg4djg/edit?usp=drive_web When you experiment with the settings,
I see that even with 1000 iterations the backprob is not capable to train the NN properly. If rprop is used it can learn decent parameters. However, it still needs 1000 iterations.
The conclusion is that this is a hard task.
Still, it would be nice to be able to train a decent NN in theanets for this XOR classifier.
Maybe RPROP trainer would a nice to have feature in theanets.
Best regards, Filip Jurcicek
Work tel. (CZ): +420221914402 Personal tel. (CZ): +420777805048 Skype: bozskyfilip
http://ufal.mff.cuni.cz/staff.html http://sites.google.com/site/filipjurcicek/
On 5 September 2013 04:47, Leif Johnson notifications@github.com wrote:
Today I verified that the neural network is indeed capable of _computing_the XOR function. In your example script, I added the following lines between creating the Experiment and calling .run():
e.network.weights[0].set_value(np.array([[100, 100], [100, 100]])) e.network.biases[0].set_value(np.array([-70, -130])) e.network.weights[1].set_value(np.array([[1], [-1]])) e.network.biases[1].set_value(np.array([0]))
(Also, make sure that you are using layers=(2, 2, 1).)
This sets the parameters of the network to fixed values that will compute the XOR function, and it does actually work.
The next question to address is, why is the optimization procedure not finding these (or equivalent) weights ? It's a very difficult optimization problem, since there are (in this network) 11 unknowns, but only 4 equations. At any rate, I'll have a look at it in the next few days.
— Reply to this email directly or view it on GitHubhttps://github.com/lmjohns3/theano-nets/pull/9#issuecomment-23840721 .
Thanks for checking on that, Filip -- it's a relief to know pybrain isn't using some magic backprop implementation to train this network.
Indeed, it would be great to add an implementation of RPROP to trainer.py ! Feel free to send a pull request if you work one up.
I may take a stab at this, unless someone of beats me to it.
The code is pretty clearly spelled out here - however it is a batch algorithm only from what I can tell.
Hi,
I have seen on many occasions that RPROP works better than many other algorithms. So, it may be useful to have it in the library.
However, few days ago I tried the HF trainer, and this works fine for the XOR task. It needs only few iterations compared to 10000 iterations for SGD. It also works well on the task I am really interested which is Voice Activity Detection for my Spoken Dialogue System.
All the best.
Best regards, Filip Jurcicek
Work tel. (CZ): +420221914402 Personal tel. (CZ): +420777805048 Skype: bozskyfilip
http://ufal.mff.cuni.cz/staff.html http://sites.google.com/site/filipjurcicek/
On 13 September 2013 04:24, Kyle Kastner notifications@github.com wrote:
I may take a stab at this, unless someone of beats me to it.
The code is pretty clearly spelled out herehttp://aass.oru.se/%7Elilien/ml/seminars/2007_03_12c-Markus_Ingvarsson-RPROP.pdf- however it is a batch algorithm only from what I can tell.
— Reply to this email directly or view it on GitHubhttps://github.com/lmjohns3/theano-nets/pull/9#issuecomment-24369358 .
Hi,
I have created an XOR example. This could be useful for debugging. Surprisingly, it still does not learn the XOR function. In Pybrain, I managed to get it learn in 20 iterations.
All the best, Filip