Closed goerch closed 3 years ago
I'm not sure what your question is?
The evaluations above your ForwardDiff call are the analytical form (*) so if ForwardDiff doesn't return that, that's an issue with ForwardDiff.
(*) unless I made a mistake there in which case it would be very sneaky as the results are identical to those of sklearn as far as I know
Could I suggest you
I'd expect to see the derivative of sigmoid in the analytical form of the hessian.
I'm sorry but so far I'm not clear on what the issue is and what you'd like me to do.
If you think there's an error with the way the Gradient/Hessian is computed and that FD should be the ground truth, then please provide the norms as indicated above.
Adding the following lines
println("norm(H - Analytical) ",
norm(H - (X' * (Diagonal(R.σ.(y .* (X * θ))) * X) + λ * I)))
println("norm(H - FD.hessian) ",
norm(H - ForwardDiff.hessian(θ -> J(θ), θ)))
println("norm(Analytical - FD.hessian) ",
norm(X' * (Diagonal(R.σ.(y .* (X * θ))) * X) + λ * I -
ForwardDiff.hessian(θ -> J(θ), θ)))
results in
norm(H - Analytical) 0.0
norm(H - FD.hessian) 78.76258276024724
norm(Analytical - FD.hessian) 78.76258276024724
Adding another line
println("norm(Alternative - FD.hessian) ",
norm(X' * (Diagonal(y .* R.σ.(y .* (X * θ)) .* (1 .- R.σ.(y .* (X * θ))) .* y) * X) + λ * I -
ForwardDiff.hessian(θ -> J(θ), θ)))
it seems AD believes the hessian to be more like this:
norm(Alternative - FD.hessian) 5.273714211591725e-14
thanks, I'll check this in details and get back to you
ok after re-doing this by hand carefully, I see the mistake in
since the gradient of σ(x)
is σ(x)(σ(x)-1)
and so indeed the formula as you wrote it fixes this and matches what you get with AD.
Thanks for reporting this, I'll fix this asap
done, thanks again for reporting this mistake
I wanted to check test set "GH> LogitL2" with AD. Unfortunately ForwardDiff seems to have problems with the definition of logsigmoid which I therefore simplified to
Afterwards the modified test case
resulted in