search

novak-99 / MLPP

A library created to revitalize C++ as a machine learning front end. Per aspera ad astra.
MIT License
1.08k stars 155 forks source link

Hyperbolic Activations #7

Closed towermitten closed 2 years ago

towermitten commented 2 years ago

In the README, it looks like you've (planned on) implemented a lot of hyperbolic functions as activation functions.

If you don't mind me asking, are there specific cases where such functions with a diverging gradient (such as Cosh and Sinh) that you've implemented would be helpful? I am very curious, I haven't seen these used before. Thanks!

novak-99 commented 2 years ago

Hi, thank you for bringing this up. The hyperbolic functions which I have listed in the Deep NN section of the readme are existing functions in the Activation class, which a user can call independently on some scalar, vector, or matrix. While you are correct in that they are not used very much in ML/DL, I decided to include them as possible activations in ANNs because I have found that in some small scale cases, say the XOR dataset, they can work quite well in conjunction with the sigmoid function, and just wanted to leave the option to use them available for users.