search

lmjohns3 / theanets

Neural network toolkit for Python
http://theanets.rtfd.org
MIT License
328 stars 73 forks source link

An XOR example #9

Closed jurcicek closed 11 years ago

jurcicek commented 11 years ago

Hi,

I have created an XOR example. This could be useful for debugging. Surprisingly, it still does not learn the XOR function. In Pybrain, I managed to get it learn in 20 iterations.

All the best, Filip

lmjohns3 commented 11 years ago

This is gong to be a great example to add to the code. I think it should work with the latest code in master though -- could you try this again against master and update this PR without the revert if that works ?

jurcicek commented 11 years ago

Well, the pull request is rather messy. However, I do not know how to make it better. In short, it should just add xor_classifier.py example.

jurcicek commented 11 years ago

I cleaned up the pull request.

lmjohns3 commented 11 years ago

Today I verified that the neural network is indeed capable of computing the XOR function. In your example script, I added the following lines between creating the Experiment and calling .run():

e.network.weights[0].set_value(np.array([[100, 100], [100, 100]]))
e.network.biases[0].set_value(np.array([-70, -130]))
e.network.weights[1].set_value(np.array([[1], [-1]]))
e.network.biases[1].set_value(np.array([0]))

(Also, make sure that you are using layers=(2, 2, 1).)

This sets the parameters of the network to fixed values that will compute the XOR function, and it does actually work.

The next question to address is, why is the optimization procedure not finding these (or equivalent) weights ? It's a very difficult optimization problem, since there are (in this network) 11 unknowns, but only 4 equations. At any rate, I'll have a look at it in the next few days.

jurcicek commented 11 years ago

Hi,

I have an example of XOR NN in pybrain. Please see the attachment. ​ xor_pybrain_classifier.pyhttps://docs.google.com/file/d/0Bx01_P9NXYisRGxXUi1TaVg4djg/edit?usp=drive_web ​ When you experiment with the settings,

I see that even with 1000 iterations the backprob is not capable to train the NN properly. If rprop is used it can learn decent parameters. However, it still needs 1000 iterations.

The conclusion is that this is a hard task.

Still, it would be nice to be able to train a decent NN in theanets for this XOR classifier.

Maybe RPROP trainer would a nice to have feature in theanets.

Best regards, Filip Jurcicek

Work tel. (CZ): +420221914402 Personal tel. (CZ): +420777805048 Skype: bozskyfilip

http://ufal.mff.cuni.cz/staff.html http://sites.google.com/site/filipjurcicek/

On 5 September 2013 04:47, Leif Johnson notifications@github.com wrote:

Today I verified that the neural network is indeed capable of _computing_the XOR function. In your example script, I added the following lines between creating the Experiment and calling .run():

e.network.weights[0].set_value(np.array([[100, 100], [100, 100]])) e.network.biases[0].set_value(np.array([-70, -130])) e.network.weights[1].set_value(np.array([[1], [-1]])) e.network.biases[1].set_value(np.array([0]))

(Also, make sure that you are using layers=(2, 2, 1).)

This sets the parameters of the network to fixed values that will compute the XOR function, and it does actually work.

The next question to address is, why is the optimization procedure not finding these (or equivalent) weights ? It's a very difficult optimization problem, since there are (in this network) 11 unknowns, but only 4 equations. At any rate, I'll have a look at it in the next few days.

— Reply to this email directly or view it on GitHubhttps://github.com/lmjohns3/theano-nets/pull/9#issuecomment-23840721 .

lmjohns3 commented 11 years ago

Thanks for checking on that, Filip -- it's a relief to know pybrain isn't using some magic backprop implementation to train this network.

Indeed, it would be great to add an implementation of RPROP to trainer.py ! Feel free to send a pull request if you work one up.

kastnerkyle commented 10 years ago

I may take a stab at this, unless someone of beats me to it.

The code is pretty clearly spelled out here - however it is a batch algorithm only from what I can tell.

jurcicek commented 10 years ago

Hi,

I have seen on many occasions that RPROP works better than many other algorithms. So, it may be useful to have it in the library.

However, few days ago I tried the HF trainer, and this works fine for the XOR task. It needs only few iterations compared to 10000 iterations for SGD. It also works well on the task I am really interested which is Voice Activity Detection for my Spoken Dialogue System.

All the best.

Best regards, Filip Jurcicek

Work tel. (CZ): +420221914402 Personal tel. (CZ): +420777805048 Skype: bozskyfilip

http://ufal.mff.cuni.cz/staff.html http://sites.google.com/site/filipjurcicek/

On 13 September 2013 04:24, Kyle Kastner notifications@github.com wrote:

I may take a stab at this, unless someone of beats me to it.

The code is pretty clearly spelled out herehttp://aass.oru.se/%7Elilien/ml/seminars/2007_03_12c-Markus_Ingvarsson-RPROP.pdf- however it is a batch algorithm only from what I can tell.

— Reply to this email directly or view it on GitHubhttps://github.com/lmjohns3/theano-nets/pull/9#issuecomment-24369358 .