More advanced activation functions.

ddbourgin / numpy-ml

Machine learning, in numpy

https://numpy-ml.readthedocs.io/

GNU General Public License v3.0

15.35k stars 3.72k forks source link

Closed WuZhuoran closed 5 years ago

WuZhuoran commented 5 years ago

I will try to implement rest part of activation functions. Any better ideas for activation function?

kmario23 commented 5 years ago

@WuZhuoran Great! I think it's also worth looking into Activation_function#Comparison_of_activation_functions

ddbourgin commented 5 years ago

Awesome! Two quick notes:

Linear is just a special case of Affine (with slope=1, intercept=0), so no need to implement a separate function
Softmax actually already exists, but as a Layer rather than an activation function. I decided to do this because it is much more efficient to calculate the gradient through the activation and projection weights together rather than in sequence.

Otherwise, looks good! I'll have a look at your PR when I have a spare moment :)

WuZhuoran commented 5 years ago

@ddbourgin Cool. Fixed.

I will commit after you review the code. About the NumPy Style Python Docstrings, I will update ASAP.