search

es-ude / elastic-ai.creator

elastic ai.creator
MIT License
16 stars 2 forks source link

Swish Activation Function not monotonous increasing #299

Closed Sillobear closed 1 year ago

Sillobear commented 1 year ago

The swish activation function is not monotonously increasing but implements FPPrecomputedMonotonicIncreasingModule. image Although the implementation looks fine. See discussion in PR #281

julianhoever commented 1 year ago

It seems that the need for a monotonically increasing function is no longer present. The current FPPrecomputedMonotonicIncreasingModule could also handle non-monotonically increasing functions. I will rename this module to make it clear.