search

nicodjimenez / lstm

Minimal, clean example of lstm neural network training in python, for learning purposes.
1.75k stars 651 forks source link

Moved derivative sigmoid and tanh into seperate functions #15

Closed ScottMackay2 closed 7 years ago

ScottMackay2 commented 7 years ago

I struggled for a while understanding this part of the code:

i_input = (1. - self.state.i) * self.state.i * di 
df_input = (1. - self.state.f) * self.state.f * df 
do_input = (1. - self.state.o) * self.state.o * do 
dg_input = (1. - self.state.g ** 2) * dg

After a while puzzling at it, I was able to see that these notations included the derivative of a sigmoid and a tanh function. I think that adding the sigmoid_derivative and tanh_derivative function to the code makes it more clear for the viewer what is happening at that location (so I did exactly that). It is also consistent with the fact that there is a separate sigmoid function.