Closed RS-Coop closed 4 months ago
wsmoses
commented
4 months ago we should definitely make this work or at least give a better error message, but FYI so you're unblocked the issue is your f is type unstable as a return, due to the use of the global A. If you did something liek f(x)::Float64 = x'*A*x, I presume it owuld be resolved
RS-Coop
commented
4 months ago Thanks for the response @wsmoses. That idea makes sense. Trying it out I get an error reagrding no forward mode derivative for cblas_dcopy64_. If I try to use LinearAlgebra.dot() instead of writing out the multiplication explicitly I get a segmentation fault instead.
Originally, I was trying to compare the Enzyme Hvp with one I had written a while ago, which is how I ran into this issue. When I attempt to do this in a standalone file or in the REPL I run into the same issues discussed above with both variants. A MWE of this is included below.
I also have a variant of this test in test suite for a package of mine which can be found here (SFN.jl). If I run this with my variant, all tests pass, but if I include the Enzyme variant it causes the following:
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:8229: void GradientUtils::eraseFictiousPHIs(): Assertion `0' failed.
[61920] signal (6.-6): AbortedI wasn't sure how to reproduce the succesful test outside of the package, so apologies for no MWE.
This raises a question about why this specific test works sometimes within a test environment but not elsewhere, and why the two variants have different results?
using Enzyme: make_zero, make_zero!, autodiff, autodiff_deferred, Forward, Reverse, Active, Const, Duplicated, DuplicatedNoNeed, hvp!
function ehvp!(res::S, f::F, x::S, v::S) where {F, T<:AbstractFloat, S<:AbstractVector{T}}
make_zero!(res)
grad = make_zero(x)
autodiff(
Forward,
d -> autodiff_deferred(Reverse, f, Active, d),
Const,
DuplicatedNoNeed(Duplicated(x, grad), Duplicated(v, res))
)
return nothing
end
#define the quadratic
n = 10
A::Matrix{Float64} = randn((n,n))
# f(x::Vector{Float64})::Float64 = LA.dot(x,A,x)
f(x::Vector{Float64})::Float64 = x'*A*x
#hvp problem setup
x = rand(n)
v = rand(n)
product = (A+A')*v
res = zeros(n)
ehvp!(res, f, x, v)
println("Custom Hvp: ", res ≈ product)
hvp!(res, f, x, v)
println("Enzyme Hvp: ", res ≈ product)Should be fixed on main (and just tagged), reopen if it persists.
RS-Coop
commented
4 months ago Are you able to explain why the two variants both work, or whether they are both valid? It also seems the official Enzyme variant is a bit faster.
using Enzyme: make_zero, make_zero!, autodiff, autodiff_deferred, Forward, Reverse, Active, Const, Duplicated, DuplicatedNoNeed, hvp!
using BenchmarkTools
function ehvp!(res::S, f::F, x::S, v::S) where {F, T<:AbstractFloat, S<:AbstractVector{T}}
make_zero!(res)
grad = make_zero(x)
autodiff(
Forward,
d -> autodiff_deferred(Reverse, f, Active, d),
Const,
DuplicatedNoNeed(Duplicated(x, grad), Duplicated(v, res))
)
return nothing
end
#define the quadratic
n = 10
A::Matrix{Float64} = randn((n,n))
# f(x::Vector{Float64})::Float64 = LA.dot(x,A,x)
f(x::Vector{Float64})::Float64 = x'*A*x
#hvp problem setup
x = rand(n)
v = rand(n)
product = (A+A')*v
res = zeros(n)
ehvp1!(res, f, x, v)
println("Custom Hvp: ", res ≈ product)
println(res, '\n')
hvp!(res, f, x, v)
println("Enzyme Hvp: ", res ≈ product)
println(res, '\n')
#benchmark
println("\nBenchmarking...")
function rosenbrock(x)
res = 0.0
for i = 1:size(x,1)-1
res += 100*(x[i+1]-x[i]^2)^2 + (1-x[i])^2
end
return res
end
dim = 100
x = zeros(dim)
result = similar(x)
t = @benchmark ehvp1!($result, $rosenbrock, $x, v) setup=(v=randn(dim))
println("Custom")
display(t)
println()
t = @benchmark hvp!($result, $rosenbrock, $x, v) setup=(v=randn(dim))
println("Enzyme")
display(t)
println()Looks reasonable enough to me.
I think the reasons the builtin one may be faster is
1) you use a closure x -> autodiff_deferred.... This forces recompilation of the derivative code every time [since it doesnt get cached]
2) Doing separate arguments rather than Duplicated(Duplicated(x, y), ...) is likely faster
RS-Coop
commented
4 months ago This still incurs a seg fault when using LinearAlgebra.dot(x,A,x) instead of x'*A*x. Should I reopen or create a new issue?
wsmoses
commented
4 months ago Yeah open an issue
On Tue, Jul 9, 2024 at 7:37 PM Cooper Simpson @.***> wrote:
This still incurs a seg fault when using LinearAlgebra.dot(x,A,x) instead of x'Ax. Should I reopen or create a new issue?
— Reply to this email directly, view it on GitHub https://github.com/EnzymeAD/Enzyme.jl/issues/1597#issuecomment-2218998270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXA365O62E6QYVCOCVDZLRX45AVCNFSM6AAAAABKGQUA2SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYHE4TQMRXGA . You are receiving this because you modified the open/close state.Message ID: @.***>
Was trying out the newly added Hessian-vector product function
hvp!on a simple quadratic and I have run into the following strange error: