search

FluxML / Zygote.jl

21st century AD
https://fluxml.ai/Zygote.jl/
Other
1.48k stars 212 forks source link

nested derivative does not work #33

Open afternone opened 5 years ago

afternone commented 5 years ago 
julia> derivative(x->x*derivative(y->x+y,1),1)
ERROR: MethodError: no method matching exprtype(::Core.Compiler.IRCode, ::String)
Closest candidates are:
  exprtype(::Core.Compiler.IRCode, ::Expr) at C:\Users\han\.julia\packages\Zygote\5mNII\src\tools\ir.jl:54
  exprtype(::Core.Compiler.IRCode, ::QuoteNode) at C:\Users\han\.julia\packages\Zygote\5mNII\src\tools\ir.jl:51
  exprtype(::Core.Compiler.IRCode, ::GlobalRef) at C:\Users\han\.julia\packages\Zygote\5mNII\src\tools\ir.jl:50
  ...
Stacktrace:
 [1] _broadcast_getindex_evalf at .\broadcast.jl:574 [inlined]
 [2] _broadcast_getindex at .\broadcast.jl:547 [inlined]
 [3] getindex at .\broadcast.jl:507 [inlined]
 [4] copyto_nonleaf!(::Array{DataType,1}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Base.Broadcast.Extruded{Array{Any,1},Tuple{Bool},Tuple{Int64}}}}, ::Base.OneTo{Int64}, ::Int64, ::Int64) at .\broadcast.jl:923
 [5] copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Array{Any,1}}}) at .\broadcast.jl:786
 [6] materialize(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Array{Any,1}}}) at .\broadcast.jl:748
 [7] record!(::Core.Compiler.IRCode) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\reverse.jl:144
 [8] #Primal#39(::Int64, ::Type, ::Core.Compiler.IRCode) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\reverse.jl:189
 [9] Type at .\none:0 [inlined]
 [10] #Adjoint#65 at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\reverse.jl:387 [inlined]
 [11] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Int64}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at .\none:0
 [12] _lookup_grad(::Type) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\emit.jl:121
 [13] #s18#633 at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface2.jl:17 [inlined]
 [14] #s18#633(::Any, ::Any, ::Any) at .\none:0
 [15] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at .\boot.jl:506
 [16] derivative at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:37 [inlined]
 [17] (::Zygote.J{Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64,getfield(Zygote, Symbol("##148#back2#115")){getfield(Zygote, Symbol("##111#113")){1,Int64}},Zygote.J{Tuple{typeof(gradient),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(gradient)}}}})(::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface2.jl:0
 [18] #31 at .\REPL[11]:1 [inlined]
 [19] (::Zygote.J{Tuple{getfield(Main, Symbol("##31#33")),Int64},Tuple{getfield(Main, Symbol("##31#33")),Int64,getfield(Zygote, Symbol("##796#back2#450")){getfield(Zygote, Symbol("##448#449")){Int64,Int64}},Zygote.J{Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64,getfield(Zygote, Symbol("##148#back2#115")){getfield(Zygote, Symbol("##111#113")){1,Int64}},Zygote.J{Tuple{typeof(gradient),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(gradient)}}}},getfield(Zygote, Symbol("##194#back2#147")){Zygote.Jnew{getfield(Main, Symbol("##32#34")){Int64},Nothing}}}})(::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface2.jl:0
 [20] (::getfield(Zygote, Symbol("##66#67")){Zygote.J{Tuple{getfield(Main, Symbol("##31#33")),Int64},Tuple{getfield(Main, Symbol("##31#33")),Int64,getfield(Zygote, Symbol("##796#back2#450")){getfield(Zygote, Symbol("##448#449")){Int64,Int64}},Zygote.J{Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64,getfield(Zygote, Symbol("##148#back2#115")){getfield(Zygote, Symbol("##111#113")){1,Int64}},Zygote.J{Tuple{typeof(gradient),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(gradient)}}}},getfield(Zygote, Symbol("##194#back2#147")){Zygote.Jnew{getfield(Main, Symbol("##32#34")){Int64},Nothing}}}}})(::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:28
 [21] gradient(::Function, ::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:34
 [22] derivative(::Function, ::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:37
 [23] top-level scope at none:0
MasonProtter commented 5 years ago

Related (I think?)

pkg"add Zygote#master IRTools#master"

julia> using Zygote

julia> f(x) = 3x^2 +2x + 1
f (generic function with 1 method)

julia> f'(2)
14

julia> f''(2)
ERROR: MethodError: no method matching Zygote.Pullback{Tuple{Type{
Stacktrace:
 [1] Type at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:10 [inlined]
 [2] _forward(::Zygote.Context, ::Type{Zygote.Pullback{Tuple{typeof(Base.literal_pow),typeof(^),Int64,Val{2}},T} where T}, ::Tuple{getfield(Zygote, Symbol("##1458#back#562")){getfield(Zygote, Symbol("##560#561")){Int64,Int64}}}) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [3] literal_pow at ./intfuncs.jl:243 [inlined]
 [4] _forward(::Zygote.Context, ::typeof(Zygote._forward), ::Zygote.Context, ::typeof(Base.literal_pow), ::typeof(^), ::Int64, ::Val{2}) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [5] f at ./REPL[2]:1 [inlined]
 [6] _forward(::Zygote.Context, ::typeof(Zygote._forward), ::Zygote.Context, ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [7] _forward at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/lib.jl:74 [inlined]
 [8] _forward(::Zygote.Context, ::typeof(Zygote._forward), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [9] macro expansion at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/lib.jl:74 [inlined]
 [10] adjoint at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/grad.jl:33 [inlined]
 [11] _forward at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/grad.jl:35 [inlined]
 [12] forward at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:37 [inlined]
 [13] _forward(::Zygote.Context, ::typeof(forward), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [14] gradient at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/lib.jl:74 [inlined]
 [15] _forward(::Zygote.Context, ::typeof(gradient), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [16] derivative at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:47 [inlined]
 [17] _forward(::Zygote.Context, ::typeof(derivative), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [18] #68 at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:49 [inlined]
 [19] _forward(::Zygote.Context, ::getfield(Zygote, Symbol("##68#69")){typeof(f)}, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [20] _forward(::Function, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:31
 [21] forward(::Function, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:37
 [22] gradient(::Function, ::Int64, ::Vararg{Int64,N} where N) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:42
 [23] derivative(::getfield(Zygote, Symbol("##68#69")){typeof(f)}, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:47
 [24] (::getfield(Zygote, Symbol("##68#69")){getfield(Zygote, Symbol("##68#69")){typeof(f)}})(::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:49
 [25] top-level scope at none:0SYSTEM: show(lasterr) caused an error
ErrorException("type DataType has no field var")

Stacktrace:
 [1] Type at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:10 [inlined]
 [2] _forward(::Zygote.Context, ::Type{Zygote.Pullback{Tuple{typeof(Base.literal_pow),typeof(^),Int64,Val{2}},T} where T}, ::Tuple{getfield(Zygote, Symbol("##1458#back#562")){getfield(Zygote, Symbol("##560#561")){Int64,Int64}}}) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [3] literal_pow at ./intfuncs.jl:243 [inlined]
 [4] _forward(::Zygote.Context, ::typeof(Zygote._forward), ::Zygote.Context, ::typeof(Base.literal_pow), ::typeof(^), ::Int64, ::Val{2}) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [5] f at ./REPL[2]:1 [inlined]
 [6] _forward(::Zygote.Context, ::typeof(Zygote._forward), ::Zygote.Context, ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [7] _forward at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/lib.jl:74 [inlined]
 [8] _forward(::Zygote.Context, ::typeof(Zygote._forward), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [9] macro expansion at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/lib.jl:74 [inlined]
 [10] adjoint at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/grad.jl:33 [inlined]
 [11] _forward at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/grad.jl:35 [inlined]
 [12] forward at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:37 [inlined]
 [13] _forward(::Zygote.Context, ::typeof(forward), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [14] gradient at /Users/mason/.julia/packages/Zygote/64nLx/src/lib/lib.jl:74 [inlined]
 [15] _forward(::Zygote.Context, ::typeof(gradient), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [16] derivative at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:47 [inlined]
 [17] _forward(::Zygote.Context, ::typeof(derivative), ::typeof(f), ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [18] #68 at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:49 [inlined]
 [19] _forward(::Zygote.Context, ::getfield(Zygote, Symbol("##68#69")){typeof(f)}, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface2.jl:0
 [20] _forward(::Function, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:31
 [21] forward(::Function, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:37
 [22] gradient(::Function, ::Int64, ::Vararg{Int64,N} where N) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:42
 [23] derivative(::getfield(Zygote, Symbol("##68#69")){typeof(f)}, ::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:47
 [24] (::getfield(Zygote, Symbol("##68#69")){getfield(Zygote, Symbol("##68#69")){typeof(f)}})(::Int64) at /Users/mason/.julia/packages/Zygote/64nLx/src/compiler/interface.jl:49
 [25] top-level scope at none:0
 [26] eval(::Module, ::Any) at ./boot.jl:319
 [27] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/mason/julia/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:85
 [28] macro expansion at /Users/mason/julia/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:117 [inlined]
 [29] (::getfield(REPL, Symbol("##26#27")){REPL.REPLBackend})() at ./task.jl:259
Goysa2 commented 5 years ago

Has this been corrected? Is there any plans to support nested differentiation in the (near) future of Zygote?

I have encountered a very similar problem this morning.

MikeInnes commented 5 years ago

Some simple things might work, it worked well at some point but has bit rotted. It's not a really high priority but I would like to revive it in general.

woct0rdho commented 5 years ago

Hi there, I'm actively following this feature. Our project involves taking nested derivatives into neural ODEs, while using the adjoint method to reduce memory usage. However, no autodiff framework can easily achieve this AFAIK. I feel that source-to-source autodiff is more efficient than tape-based autodiff when calculating nested derivatives of a same function for many times. It would be nice if you could support this feature.

adam-coogan commented 5 years ago

I'm also wondering about this -- I need to take second derivatives of functions for my work. I'm trying to use hessian instead, but unsurprisingly that doesn't work and gives the following error:

ERROR: Need an adjoint for constructor ForwardDiff.Dual{Nothing,Float64,1}. Gradient is of type ForwardDiff.Dual{Nothing,Float64,1}

I was unaware Zygote uses ForwardDiff behind the scenes. Is switching to using ForwardDiff the current workaround?

MikeInnes commented 5 years ago

For hessians specifically, we use forward mode over Zygote's reverse, since this is an efficient option (regardless of Zygote's nested AD not being supported). It looks like our current implementation has bitrotted a bit, but it may be fairly easy to fix.