Add support for "fast" hyperbolic functions

[mBarreau]

Hej,

The support of functions such as tanh_fast or sigmoid_fast is missing. I tried to add it but it seems not that easy (could it be possible to overload tanh_fast == tanh on a symbolic? Would it be somehow possible to support all activation functions in NNlib?

[tansongchen]

I also noticed that not supporting functions like sigmoid_fast caused error when differentiating through NNs involving these activation functions. However, if we directly add NNlib as a dependency, as we did for special functions, we will soon add a great amount of dependencies (let's call them function libraries) that the user of this package might be unhappy.

Other packages, like Zygote and ChainRules, specify that the rules should be a part of a function library, instead of a AD library. This could be a solution to us.

Another solution would be using new features in Pkg.jl like weak dependencies or conditional loading. This might be more feasible since we are so far only an experimental AD library.

That being said, I still need to do some experiments to find the correct way of soving this. It's great that you raised the question and I welcome more thoughts on this!

[mBarreau]

I don't really know, I tried to add NNlib and changed the code in codegen but it did not make it work. This tanh_fast seems to be a weird function and does not behave normally. I can try to send you a code, you might see the problem directly.

Otherwise, I have never used the new features in Pkg.jl and it might be a good thing to try!

[tansongchen]

Oh ok, you can either paste the code here in the issue, or open a new PR for that. I will try to solve this based on your current progress.

[vpuri3]

You can add add these methods in TaylorDiffNNlibExt or NNlibTaylorDiffExt like here:

https://github.com/YichengDWu/Sophon.jl/blob/main/ext/SophonTaylorDiffExt.jl#L16-L24

[tansongchen]

Hi @vpuri3 @mBarreau , activation functions from NNlib have been added to TaylorDiff at #67

[mBarreau]

Good job!