search

HuwCampbell / grenade

Deep Learning in Haskell
BSD 2-Clause "Simplified" License
1.44k stars 84 forks source link

Added the sinusoid activation function layer #50

Closed schnecki closed 6 years ago

HuwCampbell commented 6 years ago

Code looks good.

What's the purpose of a sinusoidal activation function though for deep learning? I feel I'm missing something. Could you please explain why you think this is important?

Also, the module name in the docs say tanh.

HuwCampbell commented 6 years ago

Thank you for contributing! It's very much appreciated.

schnecki commented 6 years ago

Well I use it as the output layer in an reinforcement learning task with output [-1;1]. This ensures that the output is in that range, regardless of the input the the function. However, there are other applications (more appropriate ones I'd say), e.g. see [1]. They predict a seasonal pattern, for which the sin-function makes sense.

[1] Gashler, Michael S., and Stephen C. Ashmore. "Training deep fourier neural networks to fit time-series data." International Conference on Intelligent Computing. Springer, Cham, 2014. Online available at https://arxiv.org/abs/1405.2262.