search

microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.5k stars 4.29k forks source link

Feature Request: Add Mish activation #3774

Open digantamisra98 opened 4 years ago

digantamisra98 commented 4 years ago

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10: (Better accuracy and faster than GELU) se50_1

haixpham commented 4 years ago

@delzac has this activation function in his cntkx library. Check out his very cool work.

digantamisra98 commented 4 years ago

@haixpham Aware of it, even included it in my readme. Awesome work by @delzac. Hopefully can be added up to CNTK as well.