search

yixuan / MiniDNN

A header-only C++ library for deep neural networks
392 stars 92 forks source link

Feature Request: Add Mish activation #8

Closed digantamisra98 closed 4 years ago

digantamisra98 commented 4 years ago

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10: (Better accuracy and faster than GELU) se50_1

Asheeshkrsharma commented 4 years ago

I have heard a lot of good words about Mish recently. So I think it should be in the development plans (which I might undertake). Anyways, keeping this issue open for this reason.

giovastabile commented 4 years ago

I have added it already, soon I will push everything to the master branch.

giovastabile commented 4 years ago

It is already available in the master branch. Check that it is implemented correctly