search

shub-bioinfo / Machine-Learning

This folder contains various useful python scripts
2 stars 0 forks source link

Activation Functions #1

Open shub-bioinfo opened 3 months ago

shub-bioinfo commented 3 months ago
shub-bioinfo commented 3 months ago

Blog containing details of different activation functions ReLU, GeLU, GLU, Swish, SwiGLU...

https://medium.com/@tariqanwarph/activation-function-and-glu-variants-for-transformer-models-a4fcbe85323f