autodiff in default algorithm

SciML / DifferentialEquations.jl

Multi-language suite for high-performance solvers of differential equations and scientific machine learning (SciML) components. Ordinary differential equations (ODEs), stochastic differential equations (SDEs), delay differential equations (DDEs), differential-algebraic equations (DAEs), and more in Julia.

https://docs.sciml.ai/DiffEqDocs/stable/

Other

2.88k stars 229 forks source link

autodiff in default algorithm #1027

Open ArnoStrouwen opened 7 months ago

ArnoStrouwen commented 7 months ago

Is it possible to do a try/catch on the evaluation of the Jac to turn on autodiff? https://github.com/SciML/DifferentialEquations.jl/blob/master/src/ode_default_alg.jl#L49

ChrisRackauckas commented 7 months ago

try/catch always has a ton of overhead so that's not good for the default method. It only works in the adjoints because the cost of adjoint differentiation is already so high when it makes sense, so the overhead is always negligible. For many default ODEs, you'd just hit a standard explicit RK method and in those cases the try/catch would be noticeable but never used.

Maybe the right thing to do would be Core.return_type and check if it's a Union{}. That would tell you if it's provable from the types to error? https://github.com/SciML/OrdinaryDiffEq.jl/pull/2103 should merge this week, and with that we may want to reconsider how this could be done @oscardssmith

oscardssmith commented 7 months ago

I'm really not sure why you think try catch has so much overhead. I measure it as ~10ns in the case where no exception is thrown.

julia> function f(n)
           try
               return n[]+1
           catch
           end
       end
f (generic function with 1 method)

julia> @btime f(Any[100])
  34.174 ns (1 allocation: 64 bytes)
101

julia> function g(n)
           return n[]+1
       end
g (generic function with 1 method)

julia> @btime g(Any[100])
  24.751 ns (1 allocation: 64 bytes)
101

ChrisRackauckas commented 7 months ago

Oh okay, then maybe in the new form we can make the AD more dynamic? Though in future versions when using ADTypes with DI this might be an issue. @gdalle something to consider.

oscardssmith commented 7 months ago

Also worth pointing out that catching an error is pretty expensive.

julia> @btime f(Any["hi"])
  20.599 μs (3 allocations: 128 bytes)

gdalle commented 7 months ago

Oh okay, then maybe in the new form we can make the AD more dynamic? Though in future versions when using ADTypes with DI this might be an issue.