search

palle-k / DL4S

Accelerated tensor operations and dynamic neural networks based on reverse mode automatic differentiation for every device that can run Swift - from watchOS to Linux
https://palle-k.github.io/DL4S/
MIT License
102 stars 13 forks source link

Feature Request: Add Mish activation #2

Closed digantamisra98 closed 4 years ago

digantamisra98 commented 4 years ago

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10: se50_1

palle-k commented 4 years ago

Thank you for requesting this feature. I have added a few new activation functions including Mish, GELU, Swish and LiSHT.

Feel free to also contribute code in the future if you are interested.

digantamisra98 commented 4 years ago

@palle-k Thank you for the consideration. Definitely will contribute in the future.