Closed huntrar closed 8 years ago
Hello @huntrar, The derivate does not seem correct. Have you tested your commit?
My mistake, you are definitely correct. After checking it seems the derivate should be equivalent to that of the sigmoid function. I have reflected this change now.
The new derivative is only valid if combined with a cross-entropy cost function implemented with a "special case" derivative. Please refer to the image below:
While the general form and derivative of the cross entropy function is defined as:
Image credits: CSC2515 (Fall 2007) lectured by Geoff Hinton and Rich Zemel
Apologies again, it should be clear at this point I am new to this. I will work on correcting this derivative.
In the meantime, I have added MSE to cost_functions. Does it seem correct to you?
It seems incorrect actually, the error remained constant while testing.
Closing this PR, may resubmit another when I am more confident in my changes...
Added softmax activation function + derivative