Open AkchurinDA opened 5 months ago
Most special functions are only implemented in a particular precision, so you can't plug in arbitrary numeric types and expect them to work.
In principle, it should be possible to make some such methods work with a type like Dual{Float64}
that is based on the same precision, but this would need to be done on a case-by-case basis.
Just to chime in, some of these methods do work with Zygote.jl, I had some provable success using it with __gamma_inc(...)
when for the longest time I had to use finite difference. That being said, I can only get gradients with Zygote.jl, because Hessians actually use ForwardDiff.jl (using Zygote.jl on itself has some serious problems); this is a shame because most maximum likelihood problems have very few parameters and are expensive to evaluate, so being able to efficiently and reliably calculate Hessians enables much better optimization algorithms. I can still get by with finite difference methods, but being able to use ForwardDiff.jl on special functions that commonly apply to statistics (particularly anything related to Beta, and Gamma distributions, but I think Normal is covered) opens a lot of doors for very good maximum likelihood estimation solutions.
Maybe Enzyme.jl will solve these issues, but I'm not sure. However, at the time of writing, I can't even get gradients on Gamma cdfs with it.
@Deduction42 Thank you for the comment! It is great to know that you can still work it out in Zygote.jl. Would you mind providing a small MWE of how you made it work? I think it will be useful to me and people who stumble upon the issue in the future.
I used this.
using Distributions
using Zygote
f(θ) = cdf(Gamma(θ[1], θ[2]), 0.5)
∂f(θ) = gradient(f, θ)
display("Calculating gradient...")
∂f([2.0,0.5])
It's actually the first time autodiff has worked for me (last time I tried was three years ago) on this. I tend to run into Bayesian inference and maximum-likelihood estimates on Gamma and Beta distributions from time to time. Zygote has come a long way, and Enzyme is supposed to be even better when it fully arrives.
I'm trying to run sensitivity analysis using
Fortuna.jl
in the example shown here:The example fails to run due to the fact that internals of the incomplete gamma function, such as
__gamma_inc_inv()
, enforceFloat64
type and, as the result, errors onForwardDiff.jl
'sDual()
type. Here a truncated error message: