search

jorgenkg / python-neural-network

This is an efficient implementation of a fully connected neural network in NumPy. The network can be trained by a variety of learning algorithms: backpropagation, resilient backpropagation and scaled conjugate gradient learning. The network has been developed with PYPY in mind.
BSD 2-Clause "Simplified" License
297 stars 98 forks source link

Added softmax activation function #6

Closed huntrar closed 8 years ago

huntrar commented 8 years ago

Added softmax activation function + derivative

jorgenkg commented 8 years ago

Hello @huntrar, The derivate does not seem correct. Have you tested your commit?

huntrar commented 8 years ago

My mistake, you are definitely correct. After checking it seems the derivate should be equivalent to that of the sigmoid function. I have reflected this change now.

jorgenkg commented 8 years ago

The new derivative is only valid if combined with a cross-entropy cost function implemented with a "special case" derivative. Please refer to the image below:

her

While the general form and derivative of the cross entropy function is defined as:

der

Image credits: CSC2515 (Fall 2007) lectured by Geoff Hinton and Rich Zemel

huntrar commented 8 years ago

Apologies again, it should be clear at this point I am new to this. I will work on correcting this derivative.

In the meantime, I have added MSE to cost_functions. Does it seem correct to you?

huntrar commented 8 years ago

It seems incorrect actually, the error remained constant while testing.

Closing this PR, may resubmit another when I am more confident in my changes...