search

josephjaspers / blackcat_tensors

Matrix-Vector Library Designed for Neural Network Construction. cuda (gpu) support, openmp (multithreaded cpu) support, partial support of BLAS, expression template based implementation PTX code generation identical to hand written kernels, and support for auto-differentiation
12 stars 4 forks source link

Feature Request: Add Mish activation #54

Closed digantamisra98 closed 4 years ago

digantamisra98 commented 4 years ago

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10: (Better accuracy and faster than GELU) se50_1

josephjaspers commented 4 years ago

Very nice work!

Added: https://github.com/josephjaspers/blackcat_tensors/commit/87fa2f427d044a120816f2bc83176bdb455ceccd

digantamisra98 commented 4 years ago

@josephjaspers Thank you for the appraisal and the consideration!