search

EnzymeAD / Enzyme.jl

Julia bindings for the Enzyme automatic differentiator
https://enzyme.mit.edu
MIT License
443 stars 62 forks source link

Inactivated PyCall Calling Functions Block Enzyme? #1481

Closed ChrisRackauckas closed 3 months ago

ChrisRackauckas commented 4 months ago

This seems unintentional. Found this as part of a bigger push https://github.com/ODINN-SciML/ODINN.jl/pull/151. If I have these two lines:

            MB_timestep!(model, glacier, params.solver.step, integrator.t; batch_id = batch_id)
            apply_MB_mask!(integrator.u, glacier, model.iceflow[batch_id])

https://github.com/ODINN-SciML/ODINN.jl/pull/151/files#diff-71e70aa5cc3c0507c49d0979eb88a9ac88cf5771a6bfe621f134fdbe146aba34R135-R136

uncommented, I get an error due to PyCall:

ERROR: Enzyme execution failed.
Enzyme: jl_call calling convention not implemented in aug_forward for   %9 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @jl_f_finalizer, {} addrspace(10)* noundef null, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 5554339744 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull align 8 dereferenceable(8) %newstruct) #65, !dbg !81
Stacktrace:
 [1] finalizer
   @ ./gcutils.jl:87
 [2] PyObject
   @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:79
 [3] __getproperty
   @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:314
 [4] getproperty
   @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:318
 [5] get_cumulative_climate!
   @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:62

Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:1338
  [2] finalizer
    @ ./gcutils.jl:87 [inlined]
  [3] PyObject
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:79 [inlined]
  [4] __getproperty
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:314 [inlined]
  [5] getproperty
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:318 [inlined]
  [6] get_cumulative_climate! (repeats 2 times)
    @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:62
  [7] get_cumulative_climate!
    @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:0 [inlined]
  [8] augmented_julia_get_cumulative_climate__24808_inner_1wrap
    @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:0
  [9] macro expansion
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5916 [inlined]
 [10] enzyme_call
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5566 [inlined]
 [11] AugmentedForwardThunk
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5454 [inlined]
 [12] runtime_generic_augfwd(activity::Type{…}, width::Val{…}, ModifiedBetween::Val{…}, RT::Val{…}, f::typeof(get_cumulative_climate!), df::Nothing, primal_1::Climate2D{…}, shadow_1_1::Climate2D{…}, primal_2::StepRange{…}, shadow_2_1::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/rules/jitrules.jl:179
 [13] #MB_timestep!#3
    @ ~/.julia/dev/Muninn/src/models/mass_balance/mass_balance_utils.jl:23
 [14] MB_timestep!
    @ ~/.julia/dev/Muninn/src/models/mass_balance/mass_balance_utils.jl:19
 [15] action!
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:135
 [16] #273
    @ ~/.julia/dev/SciMLSensitivity/src/callback_tracking.jl:388
 [17] #136
    @ ~/.julia/dev/SciMLSensitivity/src/adjoint_common.jl:439 [inlined]
 [18] diffejulia__136_19143_inner_1wrap
    @ ~/.julia/dev/SciMLSensitivity/src/adjoint_common.jl:0
 [19] macro expansion
    @ ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5916 [inlined]
 [20] enzyme_call
    @ ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5566 [inlined]
 [21] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5443 [inlined]
 [22] autodiff
    @ ~/.julia/packages/Enzyme/F71IJ/src/Enzyme.jl:291 [inlined]
 [23] _vecjacobian!(dλ::Vector{…}, y::Matrix{…}, λ::Vector{…}, p::ComponentArrays.ComponentVector{…}, t::Float64, S::SciMLSensitivity.CallbackSensitivityFunction{…}, isautojacvec::SciMLSensitivity.EnzymeVJP, dgrad::Nothing, dy::Nothing, W::Nothing)
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/derivative_wrappers.jl:696
 [24] #vecjacobian!#18
    @ ~/.julia/dev/SciMLSensitivity/src/derivative_wrappers.jl:231 [inlined]
 [25] (::SciMLSensitivity.var"#affect!#272"{…})(integrator::OrdinaryDiffEq.ODEIntegrator{…})
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/callback_tracking.jl:338
 [26] (::DiffEqCallbacks.var"#112#116"{…})(c::SciMLBase.DiscreteCallback{…}, u::Vector{…}, t::Float64, integrator::OrdinaryDiffEq.ODEIntegrator{…})
    @ DiffEqCallbacks ~/.julia/packages/DiffEqCallbacks/9fKPq/src/preset_time.jl:75
 [27] initialize!
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/callbacks.jl:13 [inlined]
 [28] initialize! (repeats 2 times)
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/callbacks.jl:14 [inlined]
 [29] initialize!
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/callbacks.jl:7 [inlined]
 [30] initialize_callbacks!(integrator::OrdinaryDiffEq.ODEIntegrator{…}, initialize_save::Bool)
    @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:663
 [31] __init(prob::SciMLBase.ODEProblem{…}, alg::OrdinaryDiffEq.RDPK3Sp35{…}, timeseries_init::Tuple{}, ts_init::Tuple{}, ks_init::Tuple{}, recompile::Type{…}; saveat::Vector{…}, tstops::Vector{…}, d_discontinuities::Tuple{}, save_idxs::Nothing, save_everystep::Bool, save_on::Bool, save_start::Bool, save_end::Bool, callback::SciMLBase.CallbackSet{…}, dense::Bool, calck::Bool, dt::Float64, dtmin::Float64, dtmax::Float64, force_dtmin::Bool, adaptive::Bool, gamma::Rational{…}, abstol::Float64, reltol::Float64, qmin::Rational{…}, qmax::Int64, qsteady_min::Int64, qsteady_max::Int64, beta1::Nothing, beta2::Nothing, qoldinit::Rational{…}, controller::Nothing, fullnormalize::Bool, failfactor::Int64, maxiters::Int64, internalnorm::typeof(DiffEqBase.ODE_DEFAULT_NORM), internalopnorm::typeof(LinearAlgebra.opnorm), isoutofdomain::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), unstable_check::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), verbose::Bool, timeseries_errors::Bool, dense_errors::Bool, advance_to_tstop::Bool, stop_at_next_tstop::Bool, initialize_save::Bool, progress::Bool, progress_steps::Int64, progress_name::String, progress_message::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), progress_id::Symbol, userdata::Nothing, allow_extrapolation::Bool, initialize_integrator::Bool, alias_u0::Bool, alias_du0::Bool, initializealg::OrdinaryDiffEq.DefaultInit, kwargs::@Kwargs{})
    @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:517
 [32] __init(prob::Union{…}, alg::Union{…}, timeseries_init::Any, ts_init::Any, ks_init::Any, recompile::Type{…}) where recompile_flag (repeats 5 times)
    @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:11 [inlined]
 [33] #__solve#805
    @ ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:6 [inlined]
 [34] __solve
    @ ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:1 [inlined]
 [35] solve_call(_prob::SciMLBase.ODEProblem{…}, args::OrdinaryDiffEq.RDPK3Sp35{…}; merge_callbacks::Bool, kwargshandle::Nothing, kwargs::@Kwargs{…})
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:612
 [36] solve_call
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:569 [inlined]
 [37] #solve_up#53
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1080 [inlined]
 [38] solve_up
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1066 [inlined]
 [39] #solve#51
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1003 [inlined]
 [40] _adjoint_sensitivities(sol::SciMLBase.ODESolution{…}, sensealg::SciMLSensitivity.GaussAdjoint{…}, alg::OrdinaryDiffEq.RDPK3Sp35{…}; t::Vector{…}, dgdu_discrete::Function, dgdp_discrete::Nothing, dgdu_continuous::Nothing, dgdp_continuous::Nothing, g::Nothing, abstol::Float64, reltol::Float64, checkpoints::Vector{…}, corfunc_analytical::Bool, callback::SciMLBase.CallbackSet{…}, kwargs::@Kwargs{…})
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/gauss_adjoint.jl:540
 [41] _adjoint_sensitivities
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/gauss_adjoint.jl:507 [inlined]
 [42] #adjoint_sensitivities#63
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/sensitivity_interface.jl:386 [inlined]
 [43] (::SciMLSensitivity.var"#adjoint_sensitivity_backpass#314"{…})(Δ::SciMLBase.ODESolution{…})
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/concrete_solve.jl:556
 [44] ZBack
    @ ~/.julia/packages/Zygote/nsBv0/src/compiler/chainrules.jl:211 [inlined]
 [45] (::Zygote.var"#kw_zpullback#53"{…})(dy::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/chainrules.jl:237
 [46] #291
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [47] (::Zygote.var"#2169#back#293"{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [48] #solve#51
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1003 [inlined]
 [49] (::Zygote.Pullback{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [50] #291
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [51] (::Zygote.var"#2169#back#293"{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [52] solve
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:993 [inlined]
 [53] (::Zygote.Pullback{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [54] #simulate_iceflow_UDE!#29
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:187 [inlined]
 [55] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Nothing)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [56] simulate_iceflow_UDE!
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:171 [inlined]
 [57] (::Zygote.Pullback{Tuple{…}, Any})(Δ::Nothing)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [58] batch_iceflow_UDE
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:146 [inlined]
 [59] (::Zygote.Pullback{…})(Δ::@NamedTuple{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [60] #25
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:114 [inlined]
 [61] (::Zygote.Pullback{…})(Δ::@NamedTuple{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [62] #680
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/array.jl:201 [inlined]
 [63] #235
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:157 [inlined]
 [64] (::Base.var"#1023#1028"{Distributed.var"#235#236"{…}})(r::Base.RefValue{Any}, args::Tuple{Tuple{…}})
    @ Base ./asyncmap.jl:94
Stacktrace:
  [1] (::Base.var"#1033#1035")(x::Task)
    @ Base ./asyncmap.jl:171
  [2] foreach(f::Base.var"#1033#1035", itr::Vector{Any})
    @ Base ./abstractarray.jl:3094
  [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::Base.Iterators.Zip{Tuple{…}})
    @ Base ./asyncmap.jl:171
  [4] wrap_n_exec_twice
    @ ./asyncmap.jl:147 [inlined]
  [5] #async_usemap#1018
    @ ./asyncmap.jl:97 [inlined]
  [6] kwcall(::NamedTuple, ::typeof(Base.async_usemap), f::Any, c::Vararg{Any})
    @ Base ./asyncmap.jl:78 [inlined]
  [7] #asyncmap#1017
    @ ./asyncmap.jl:75 [inlined]
  [8] asyncmap
    @ ./asyncmap.jl:74 [inlined]
  [9] pmap(f::Function, p::Distributed.WorkerPool, c::Base.Iterators.Zip{…}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{…}, retry_check::Nothing)
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:126
 [10] pmap(f::Function, p::Distributed.WorkerPool, c::Base.Iterators.Zip{Tuple{Vector{Tuple{…}}, Vector{@NamedTuple{…}}}})
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:99
 [11] pmap(f::Function, c::Base.Iterators.Zip{Tuple{Vector{Tuple{…}}, Vector{@NamedTuple{…}}}}; kwargs::@Kwargs{})
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:156
 [12] pmap
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:156 [inlined]
 [13] pmap(f::Function, c1::Vector{Tuple{…}}, c::Vector{@NamedTuple{…}})
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:157
 [14] (::Zygote.var"#map_back#682"{ODINN.var"#25#26"{…}, 1, Tuple{…}, Tuple{…}, Vector{…}})(Δ::Vector{@NamedTuple{…}})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/array.jl:201
 [15] (::Zygote.var"#2861#back#688"{Zygote.var"#map_back#682"{…}})(Δ::Vector{@NamedTuple{…}})
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [16] predict_iceflow!
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:114 [inlined]
 [17] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Nothing)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [18] loss_iceflow
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:58 [inlined]
 [19] (::Zygote.Pullback{Tuple{…}, Any})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [20] #22
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:31 [inlined]
 [21] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [22] #291
    @ ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [23] #2169#back
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72 [inlined]
 [24] OptimizationFunction
    @ ~/.julia/packages/SciMLBase/JUp1I/src/scimlfunctions.jl:3762 [inlined]
 [25] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [26] #291
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [27] (::Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{…}, Zygote.Pullback{…}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [28] #37
    @ ~/.julia/dev/OptimizationBase/ext/OptimizationZygoteExt.jl:90 [inlined]
 [29] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [30] #291
    @ ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [31] #2169#back
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72 [inlined]
 [32] #39
    @ ~/.julia/dev/OptimizationBase/ext/OptimizationZygoteExt.jl:93 [inlined]
 [33] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [34] (::Zygote.var"#75#76"{Zygote.Pullback{Tuple{…}, Tuple{…}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface.jl:91
 [35] gradient(f::Function, args::ComponentArrays.ComponentVector{Float64, Vector{Float64}, Tuple{ComponentArrays.Axis{…}}})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface.jl:148
 [36] (::OptimizationZygoteExt.var"#38#56"{…})(::ComponentArrays.ComponentVector{…}, ::ComponentArrays.ComponentVector{…}, ::Vector{…}, ::Vararg{…})
    @ OptimizationZygoteExt ~/.julia/dev/OptimizationBase/ext/OptimizationZygoteExt.jl:93
 [37] macro expansion
    @ ~/.julia/packages/OptimizationOptimisers/AOkbT/src/OptimizationOptimisers.jl:68 [inlined]
 [38] macro expansion
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328 [inlined]
 [39] (::OptimizationOptimisers.var"#12#13"{OptimizationBase.OptimizationCache{…}, ComponentArrays.ComponentVector{…}})()
    @ OptimizationOptimisers ~/.julia/packages/Optimization/jWtfU/src/utils.jl:29
 [40] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:515
 [41] with_logger
    @ Base.CoreLogging ./logging.jl:627 [inlined]
 [42] maybe_with_logger(f::OptimizationOptimisers.var"#12#13"{…}, logger::LoggingExtras.TeeLogger{…})
    @ Optimization ~/.julia/packages/Optimization/jWtfU/src/utils.jl:7
 [43] macro expansion
    @ ~/.julia/packages/Optimization/jWtfU/src/utils.jl:28 [inlined]
 [44] __solve(cache::OptimizationBase.OptimizationCache{…})
    @ OptimizationOptimisers ~/.julia/packages/OptimizationOptimisers/AOkbT/src/OptimizationOptimisers.jl:66
 [45] solve!(cache::OptimizationBase.OptimizationCache{…})
    @ SciMLBase ~/.julia/packages/SciMLBase/JUp1I/src/solve.jl:188
 [46] solve(prob::SciMLBase.OptimizationProblem{…}, alg::Optimisers.Adam, args::IterTools.NCycle{…}; kwargs::@Kwargs{…})
    @ SciMLBase ~/.julia/packages/SciMLBase/JUp1I/src/solve.jl:96
 [47] train_UDE!(simulation::FunctionalInversion)
    @ ODINN ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:43
 [48] run!(simulation::FunctionalInversion)
    @ ODINN ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:11
 [49] top-level scope
    @ ./timing.jl:279
 [50] top-level scope
    @ none:1
Some type information was truncated. Use `show(err)` to see complete types.

However, both functions are inactivated at the top of the file:

https://github.com/ODINN-SciML/ODINN.jl/pull/151/files#diff-d3c9b3bf5488114c648831dcc69c1e95f51dd3e64362e32ed25631401ff945afR37-R39

I can "fix" this by commenting out those lines, and those functions have PyCall interior to them, so that makes it clear that these are blocking Enzyme from building the function. However, because those functions are inactivated, my understanding is that Enzyme shouldn't care what's inside of there for the purpose of differentiation. It seems that something in the compiler pass is getting stuck at what's going on with PyCalls when in this case it should just not care / skip over it.

wsmoses commented 4 months ago

I think the answer here is to add a cutom rule to the pyobject constructor which ensures that all the cross language GC is handled properly. Specifically: https://github.com/JuliaPy/PyCall.jl/blob/2f600fbebee50ab0672e153455e3c0fda1694fba/src/PyCall.jl#L53

Ironically we can almost certainly just copy the cuarray rule https://github.com/JuliaGPU/CUDA.jl/blob/bbfd1ad2c3c3ace2502d61c254a6a7f3c48f2197/ext/EnzymeCoreExt.jl#L107 and https://github.com/JuliaGPU/CUDA.jl/blob/bbfd1ad2c3c3ace2502d61c254a6a7f3c48f2197/ext/EnzymeCoreExt.jl#L304 and https://github.com/JuliaGPU/CUDA.jl/blob/bbfd1ad2c3c3ace2502d61c254a6a7f3c48f2197/ext/EnzymeCoreExt.jl#L331

wsmoses commented 3 months ago

Further investigation, I think the issue here is inactive wasn't being checked in the context of kwargs (and also there may have been kwargs here on the call)

Regardless, try https://github.com/EnzymeAD/Enzyme.jl/pull/1539 and also possibly doing args...; kwargs... on your inactive marker

wsmoses commented 3 months ago

@ChrisRackauckas going to go ahead and close this. please reopen if it persists