Closed digantamisra98 closed 4 years ago
Hi,
thank you for your proposal.
We currently do not have a plan to implement Mish activation function, but you can easily implement it using our currently available functions:
mish_x = x * F.tanh(F.softplus(x))
@TE-andrewshin Thank you for the response. No worries. The implementation is more than helpful!
Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:
All benchmarks, analysis and links to official package implementations can be found in this repository
It would be nice to have Mish as an option within the activation function group.
This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10: (Better accuracy and faster than GELU)