Feature Request: Add Mish activation

[digantamisra98]

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

• TensorFlow-Addons
• SpaCy (Tok2Vec Layer)
• Thinc - SpaCy's official NLP based ML library
• Echo AI
• Eclipse's deeplearning4j
• Hasktorch
• CNTKX - Extension of Microsoft's CNTK
• FastAI-Dev
• Darknet
• Yolov3
• BeeDNN - Library in C++
• Gen-EfficientNet-PyTorch
• dnet

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10: (Better accuracy and faster than GELU)

(It has also shown considerable improvement over GELU)

[rsdubtso]

Hi! Thanks for the suggested enhancement.

Since the activation seems to be already supported by various framework extensions packages, it does not seem urgent to be adding support to DNNL which is typically not used standalone but as a compute library underlying the frameworks (which also would mean that support for Mish would need to be upstreamed to the core of those frameworks).

I'm leaving this open for the other team members to weigh in, but my resolution that this can certainly wait until there is more customer interest in the feature.

[digantamisra98]

@rsdubtso Thank you for the response. I completely understand your POV. And agreed, it isn't a priority ofcourse. I am trying to get it added to the core of different frameworks. If you have any feedback on Mish itself, feel free to let me know. Thanks!

[vpirogov]

I'm closing this request for now as there are no plans to implement this feature y the project core team.