search

devonlee111 / Basic-Neural-Net

A basic neural net built from scratch.
MIT License
0 stars 0 forks source link

More Than One Hidden Layer Does Not Work #1

Closed devonlee111 closed 6 years ago

devonlee111 commented 6 years ago

Problem

The NN.py program does not train properly when using 2 or more hidden layers.

Possible Errors

The cause of this problem is currently unknown.

  1. Most likely a problem with the backpropagation algorithm.
    1. Backpropagation function may not be implemented fully/completely correct.
      1. Error is most likely a problem where the layer error is incorrectly calculated.
    2. There may be a minor error in the code for the backpropagation function.
  2. Potentially a slight code/logic error in forward pass method.

Methods To Find A Solution

More research will be done on the backpropagation function to see what may have been implemented incorrectly. A review of the code will be performed in case it turns out to be a minor algorithm error.

devonlee111 commented 6 years ago

Solution

It seems this problem wasn't really an implementation problem at all. It seems that the real issue was using the sigmoid function as an activation function. When using sigmoid on a neural net with more than a single hidden layer, the neural net exhibited a severe case of, what I believe to be, vanishing gradients. I have solved this problem by implementing the option of using tanh as the activation function. Using tanh seems to have removed the vanishing gradients problem when using multiple hidden layers.