Feature Request: Add Mish activation

[digantamisra98]

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

• TensorFlow-Addons
• SpaCy (Tok2Vec Layer)
• Thinc - SpaCy's official NLP based ML library
• Echo AI
• CNTKX - Extension of Microsoft's CNTK
• FastAI-Dev
• Darknet
• Yolov3
• BeeDNN - Library in C++
• Gen-EfficientNet-PyTorch
• dnet

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10:

[dfalbel]

Hi! @digantamisra98 this is very interesting!

Do you want to contribute a Pull Request implementing it as a layer_activation_mish?

[digantamisra98]

@dfalbel will need some help, still a novice in R. But if you can chime in and do a PR, that would be very helpful. Thanks

[dfalbel]

Ok! I'll leave it open as an enhancement. I have some other higher priority issues but will come back if I have time.

[digantamisra98]

@dfalbel alright. Thanks for the consideration. Meanwhile I'll try to put up a PR

[amrrs]

@dfalbel I can give a try, Do you have any example PR to refer of what has to be changed in that case?

[digantamisra98]

@dfalbel do you have any suggestion of how to integrate Mish from TensorFlow-Addons considering R Keras is using TF.Keras under the hood if I'm correct?

[dfalbel]

@digantamisra98 you can check if the module is installed and load it or return an error otherwise. for example:

if (reticulate::py_module_available("tfaddons")) {
  tfaddon <- reticulate::import("tfaddon")
} else {
  stop("Needs tfaddon. Please install with `reticulate::py_install('tfaddon')`")
}

Then you can just use the tfaddon.

@amrrs the closes PR I found is this: https://github.com/rstudio/keras/pull/838/files#diff-7089c0b7d945592843f01e094a15f211R35

I ended up moving this to a different package, but the idea is the same.

[arunamalla]

@dfalbel is this still an issue? I would like to implement if it is.

[t-kalinowski]

keras3::activation_mish() is available. Please give it a try:

remotes::install_github("rstudio/keras")
library(keras3)