Feature request: Support Sigmoid activation function

mapillary / inplace_abn

In-Place Activated BatchNorm for Memory-Optimized Training of DNNs

BSD 3-Clause "New" or "Revised" License

1.32k stars 187 forks source link

Feature request: Support Sigmoid activation function #214

Closed jasam-sheja closed 2 years ago

jasam-sheja commented 2 years ago

The sigmoid function is used in many networks, and it is beneficial to have an activated batch norm with the sigmoid function. Sigmoid is also invertible. If

$y = 1/(1+e^{-x})$

then

$x = ln(y/(1-y)) = -ln((1-y)/y)$

and

$dy/dx = y(1-y)$

If you think this is a good idea, I'd like to contribute it to this repo.

Thanks :)

jasam-sheja commented 2 years ago

Never mind, the sigmoid inverse is numerically unstable for positive numbers. This use case is limited to inputs restricted to be smaller than some positive number under 100 depending on precision. Unless there is a stable inverse for sigmoid, this won't be useful.