search

EnzymeAD / Enzyme.jl

Julia bindings for the Enzyme automatic differentiator
https://enzyme.mit.edu
MIT License
458 stars 65 forks source link

Turing model crashes without runtimeActivity #1623

Open mhauru opened 4 months ago

mhauru commented 4 months ago 
module MWE

import LogDensityProblems
using Turing: @model, MvNormal, LogDensityFunction

using Enzyme

@model function satellite_model_matrix(::Type{TV}=Matrix{Float64}) where {TV}
    @show TV
    P0 = vcat([0.1 0.0], [0.0 0.1])
    x = TV(undef, 2, 2)
    fill!(x, zero(eltype(x)))
    x[:, 2] ~ MvNormal(x[:, 1], P0)
    return nothing
end

model = satellite_model_matrix()

f = LogDensityFunction(model)
x = [1.0, 1.0]
Enzyme.autodiff(ReverseWithPrimal, LogDensityProblems.logdensity, Active, Const(f), Enzyme.Duplicated(x, zero(x)))

end
ERROR: Enzyme execution failed.
Mismatched activity for:   store {} addrspace(10)* %.fca.0.0.0.0.extract, {} addrspace(10)** %.fca.0.0.0.gep, align 8, !dbg !253, !noalias !240 const val:   %.fca.0.0.0.0.extract = extractvalue { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } %0, 0, 0, 0, 0, !dbg !97
Type tree: {[-1]:Pointer, [-1,0]:Pointer, [-1,0,0]:Pointer, [-1,0,0,-1]:Integer, [-1,0,8]:Integer, [-1,0,9]:Integer, [-1,0,10]:Integer, [-1,0,11]:Integer, [-1,0,12]:Integer, [-1,0,13]:Integer, [-1,0,14]:Integer, [-1,0,15]:Integer, [-1,0,16]:Integer, [-1,0,17]:Integer, [-1,0,18]:Integer, [-1,0,19]:Integer, [-1,0,20]:Integer, [-1,0,21]:Integer, [-1,0,22]:Integer, [-1,0,23]:Integer, [-1,0,24]:Integer, [-1,0,25]:Integer, [-1,0,26]:Integer, [-1,0,27]:Integer, [-1,0,28]:Integer, [-1,0,29]:Integer, [-1,0,30]:Integer, [-1,0,31]:Integer, [-1,0,32]:Integer, [-1,0,33]:Integer, [-1,0,34]:Integer, [-1,0,35]:Integer, [-1,0,36]:Integer, [-1,0,37]:Integer, [-1,0,38]:Integer, [-1,0,39]:Integer, [-1,8]:Pointer, [-1,8,0]:Pointer, [-1,8,0,-1]:Integer, [-1,8,8]:Integer, [-1,8,9]:Integer, [-1,8,10]:Integer, [-1,8,11]:Integer, [-1,8,12]:Integer, [-1,8,13]:Integer, [-1,8,14]:Integer, [-1,8,15]:Integer, [-1,8,16]:Integer, [-1,8,17]:Integer, [-1,8,18]:Integer, [-1,8,19]:Integer, [-1,8,20]:Integer, [-1,8,21]:Integer, [-1,8,22]:Integer, [-1,8,23]:Integer, [-1,8,24]:Integer, [-1,8,25]:Integer, [-1,8,26]:Integer, [-1,8,27]:Integer, [-1,8,28]:Integer, [-1,8,29]:Integer, [-1,8,30]:Integer, [-1,8,31]:Integer, [-1,8,32]:Integer, [-1,8,33]:Integer, [-1,8,34]:Integer, [-1,8,35]:Integer, [-1,8,36]:Integer, [-1,8,37]:Integer, [-1,8,38]:Integer, [-1,8,39]:Integer, [-1,16]:Pointer, [-1,16,0]:Pointer, [-1,16,0,-1]:Integer, [-1,16,8]:Integer, [-1,16,9]:Integer, [-1,16,10]:Integer, [-1,16,11]:Integer, [-1,16,12]:Integer, [-1,16,13]:Integer, [-1,16,14]:Integer, [-1,16,15]:Integer, [-1,16,16]:Integer, [-1,16,17]:Integer, [-1,16,18]:Integer, [-1,16,19]:Integer, [-1,16,20]:Integer, [-1,16,21]:Integer, [-1,16,22]:Integer, [-1,16,23]:Integer, [-1,16,24]:Integer, [-1,16,25]:Integer, [-1,16,26]:Integer, [-1,16,27]:Integer, [-1,16,28]:Integer, [-1,16,29]:Integer, [-1,16,30]:Integer, [-1,16,31]:Integer, [-1,16,32]:Integer, [-1,16,33]:Integer, [-1,16,34]:Integer, [-1,16,35]:Integer, [-1,16,36]:Integer, [-1,16,37]:Integer, [-1,16,38]:Integer, [-1,16,39]:Integer, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Integer, [-1,41]:Integer, [-1,42]:Integer, [-1,43]:Integer, [-1,44]:Integer, [-1,45]:Integer, [-1,46]:Integer, [-1,47]:Integer, [-1,48]:Integer, [-1,49]:Integer, [-1,50]:Integer, [-1,51]:Integer, [-1,52]:Integer, [-1,53]:Integer, [-1,54]:Integer, [-1,55]:Integer, [-1,56]:Integer, [-1,57]:Integer, [-1,58]:Integer, [-1,59]:Integer, [-1,60]:Integer, [-1,61]:Integer, [-1,62]:Integer, [-1,63]:Integer}
 llvalue=  %.fca.0.0.0.0.extract = extractvalue { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } %0, 0, 0, 0, 0, !dbg !97
You may be using a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Activity-of-temporary-storage). If not, please open an issue, and either rewrite this variable to not be conditionally active or use Enzyme.API.runtimeActivity!(true) as a workaround for now

Stacktrace:
 [1] _evaluate!!
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:967
 [2] evaluate_threadunsafe!!
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941
 [3] evaluate!!
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894
 [4] logdensity
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100
 [5] logdensity
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0

Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:1623
  [2] _evaluate!!
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:967 [inlined]
  [3] evaluate_threadunsafe!!
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941 [inlined]
  [4] evaluate!!
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894 [inlined]
  [5] logdensity
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100 [inlined]
  [6] logdensity
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0 [inlined]
  [7] diffejulia_logdensity_3364_inner_1wrap
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0
  [8] macro expansion
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6622 [inlined]
  [9] enzyme_call
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6223 [inlined]
 [10] CombinedAdjointThunk
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6100 [inlined]
 [11] autodiff
    @ ~/.julia/dev/Enzyme/src/Enzyme.jl:309 [inlined]
 [12] autodiff(::EnzymeCore.ReverseMode{true, EnzymeCore.FFIABI, false}, ::typeof(LogDensityProblems.logdensity), ::Type{EnzymeCore.Active}, ::EnzymeCore.Const{DynamicPPL.LogDensityFunction{…}}, ::EnzymeCore.Duplicated{Vector{…}})
    @ Enzyme ~/.julia/dev/Enzyme/src/Enzyme.jl:321
 [13] top-level scope
    @ REPL[1]:21
Some type information was truncated. Use `show(err)` to see complete types.

Works with runtimeActivity enabled. Enzyme_jll v0.0.133. First discussed in TuringLang/Turing.jl#1608.

wsmoses commented 4 months ago

Further reduction would be helpful here.

In essence the issue is a constant variable is being stored into a differentiable struct or something

On Wed, Jul 10, 2024 at 4:20 AM Markus Hauru @.***> wrote:

module MWE

import LogDensityProblems using Turing: @model, MvNormal, LogDensityFunction

using Enzyme

@model function satellite_model_matrix(::Type{TV}=Matrix{Float64}) where {TV} @show TV P0 = vcat([0.1 0.0], [0.0 0.1]) x = TV(undef, 2, 2) fill!(x, zero(eltype(x))) x[:, 2] ~ MvNormal(x[:, 1], P0) return nothing end

model = satellite_model_matrix()

f = LogDensityFunction(model) x = [1.0, 1.0] Enzyme.autodiff(ReverseWithPrimal, LogDensityProblems.logdensity, Active, Const(f), Enzyme.Duplicated(x, zero(x)))

end

ERROR: Enzyme execution failed. Mismatched activity for: store {} addrspace(10)* %.fca.0.0.0.0.extract, {} addrspace(10)* %.fca.0.0.0.gep, align 8, !dbg !253, !noalias !240 const val: %.fca.0.0.0.0.extract = extractvalue { { [1 x [8 x {} addrspace(10)]], {} addrspace(10), {} addrspace(10) } } %0, 0, 0, 0, 0, !dbg !97 Type tree: {[-1]:Pointer, [-1,0]:Pointer, [-1,0,0]:Pointer, [-1,0,0,-1]:Integer, [-1,0,8]:Integer, [-1,0,9]:Integer, [-1,0,10]:Integer, [-1,0,11]:Integer, [-1,0,12]:Integer, [-1,0,13]:Integer, [-1,0,14]:Integer, [-1,0,15]:Integer, [-1,0,16]:Integer, [-1,0,17]:Integer, [-1,0,18]:Integer, [-1,0,19]:Integer, [-1,0,20]:Integer, [-1,0,21]:Integer, [-1,0,22]:Integer, [-1,0,23]:Integer, [-1,0,24]:Integer, [-1,0,25]:Integer, [-1,0,26]:Integer, [-1,0,27]:Integer, [-1,0,28]:Integer, [-1,0,29]:Integer, [-1,0,30]:Integer, [-1,0,31]:Integer, [-1,0,32]:Integer, [-1,0,33]:Integer, [-1,0,34]:Integer, [-1,0,35]:Integer, [-1,0,36]:Integer, [-1,0,37]:Integer, [-1,0,38]:Integer, [-1,0,39]:Integer, [-1,8]:Pointer, [-1,8,0]:Pointer, [-1,8,0,-1]:Integer, [-1,8,8]:Integer, [-1,8,9]:Integer, [-1,8,10]:Integer, [-1,8,11]:Integer, [-1,8,12]:Integer, [-1,8,13]:Integer, [-1,8,14]:Integer, [-1,8,15]:Integer, [-1,8,16]:Integer, [-1,8,17]:Integer, [-1,8,18]:Integer, [-1,8,19]:Integer, [-1,8,20]:Integer, [-1,8,21]:Integer, [-1,8,22]:Integer, [-1,8,23]:Integer, [-1,8,24]:Integer, [-1,8,25]:Integer, [-1,8,26]:Integer, [-1,8,27]:Integer, [-1,8,28]:Integer, [-1,8,29]:Integer, [-1,8,30]:Integer, [-1,8,31]:Integer, [-1,8,32]:Integer, [-1,8,33]:Integer, [-1,8,34]:Integer, [-1,8,35]:Integer, [-1,8,36]:Integer, [-1,8,37]:Integer, [-1,8,38]:Integer, [-1,8,39]:Integer, [-1,16]:Pointer, [-1,16,0]:Pointer, [-1,16,0,-1]:Integer, [-1,16,8]:Integer, [-1,16,9]:Integer, [-1,16,10]:Integer, [-1,16,11]:Integer, [-1,16,12]:Integer, [-1,16,13]:Integer, [-1,16,14]:Integer, [-1,16,15]:Integer, [-1,16,16]:Integer, [-1,16,17]:Integer, [-1,16,18]:Integer, [-1,16,19]:Integer, [-1,16,20]:Integer, [-1,16,21]:Integer, [-1,16,22]:Integer, [-1,16,23]:Integer, [-1,16,24]:Integer, [-1,16,25]:Integer, [-1,16,26]:Integer, [-1,16,27]:Integer, [-1,16,28]:Integer, [-1,16,29]:Integer, [-1,16,30]:Integer, [-1,16,31]:Integer, [-1,16,32]:Integer, [-1,16,33]:Integer, [-1,16,34]:Integer, [-1,16,35]:Integer, [-1,16,36]:Integer, [-1,16,37]:Integer, [-1,16,38]:Integer, [-1,16,39]:Integer, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Integer, [-1,41]:Integer, [-1,42]:Integer, [-1,43]:Integer, [-1,44]:Integer, [-1,45]:Integer, [-1,46]:Integer, [-1,47]:Integer, [-1,48]:Integer, [-1,49]:Integer, [-1,50]:Integer, [-1,51]:Integer, [-1,52]:Integer, [-1,53]:Integer, [-1,54]:Integer, [-1,55]:Integer, [-1,56]:Integer, [-1,57]:Integer, [-1,58]:Integer, [-1,59]:Integer, [-1,60]:Integer, [-1,61]:Integer, [-1,62]:Integer, [-1,63]:Integer} llvalue= %.fca.0.0.0.0.extract = extractvalue { { [1 x [8 x {} addrspace(10)]], {} addrspace(10), {} addrspace(10)* } } %0, 0, 0, 0, 0, !dbg !97 You may be using a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Activity-of-temporary-storage). If not, please open an issue, and either rewrite this variable to not be conditionally active or use Enzyme.API.runtimeActivity!(true) as a workaround for now

Stacktrace: [1] _evaluate!! @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:967 [2] evaluate_threadunsafe!! @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941 [3] evaluate!! @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894 [4] logdensity @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100 [5] logdensity @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0

Stacktrace: [1] throwerr(cstr::Cstring) @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:1623 [2] _evaluate!! @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:967 [inlined] [3] evaluate_threadunsafe!! @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941 [inlined] [4] evaluate!! @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894 [inlined] [5] logdensity @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100 [inlined] [6] logdensity @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0 [inlined] [7] diffejulia_logdensity_3364_inner_1wrap @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0 [8] macro expansion @ ~/.julia/dev/Enzyme/src/compiler.jl:6622 [inlined] [9] enzyme_call @ ~/.julia/dev/Enzyme/src/compiler.jl:6223 [inlined] [10] CombinedAdjointThunk @ ~/.julia/dev/Enzyme/src/compiler.jl:6100 [inlined] [11] autodiff @ ~/.julia/dev/Enzyme/src/Enzyme.jl:309 [inlined] [12] autodiff(::EnzymeCore.ReverseMode{true, EnzymeCore.FFIABI, false}, ::typeof(LogDensityProblems.logdensity), ::Type{EnzymeCore.Active}, ::EnzymeCore.Const{DynamicPPL.LogDensityFunction{…}}, ::EnzymeCore.Duplicated{Vector{…}}) @ Enzyme ~/.julia/dev/Enzyme/src/Enzyme.jl:321 [13] top-level scope @ REPL[1]:21 Some type information was truncated. Use show(err) to see complete types.

Works with runtimeActivity enabled. Enzyme_jll v0.0.133. First discussed in TuringLang/Turing.jl#1608 https://github.com/EnzymeAD/Enzyme.jl/issues/1608.

— Reply to this email directly, view it on GitHub https://github.com/EnzymeAD/Enzyme.jl/issues/1623, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXGC3BGK4PD33GV62U3ZLTVETAVCNFSM6AAAAABKUMBRLWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYDAMBXHA4TENY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

wsmoses commented 4 months ago

bump @mhauru @yebai

yebai commented 3 months ago

Can we work around it, assuming https://github.com/TuringLang/DynamicPPL.jl/issues/643#issuecomment-2302643605 was the cause?

wsmoses commented 3 months ago

The error message thrown says to use Enzyme runtimeactivry (which allows this to run iirc).

yebai commented 3 months ago

Yes, the above model runs without issue with Enzyme.API.runtimeActivity!(true).

wsmoses commented 2 months ago

any updates on reduction here?

yebai commented 2 months ago

This is not a Turing issue, nor an Enzyme issue. See https://github.com/JuliaLang/julia/issues/55638

Also, this performance penalty is likely related to failed type inference for a customised interpreter: https://github.com/EnzymeAD/Enzyme.jl/issues/1769

wsmoses commented 2 months ago

I don't think this issue relates to that. I think we just need a reduction for a minimal code here of why runtime activity is needed.

yebai commented 2 months ago

I suggest we wait for https://github.com/JuliaLang/julia/issues/55638 to be addressed before investing any further time in this.

wsmoses commented 2 months ago

I'm not confident this is due to that. Do you have evidence or a MWE which demonstrates that?