ODE/DAE solver interface

[mauro3]

I think it would be great if all the ODE/DAE solvers could be called through the same interface. This would essentially mirror the approach of https://github.com/JuliaOpt/MathProgBase.jl . Would it be worth trying to hammer out this interface in this road-map issue?

Edit: By "same interface" I don't mean that ODE and DAE solvers have the same interface. But that all ODE solvers are wrapped to conform to the same interface (similarly all DAE, and maybe all IMEX, etc). So, Sundials.jl, ODE.jl, etc. would then depend on ODEProgBase (or probably DiffEqBase) and extend some solve function there, taking as input some problem specification type. Likewise all higher level Diff-Eq stuff (currently DifferentialEquations.jl and IVPTestSuite.jl) would program against this interface.

TODO

Specify:

• [x] solve signature: solve(p::Problem, ::Type{Alg}; kwargs...)
• [ ] Problem types
• [ ] Solution types
• [ ] default options
• for callback options see #10

[mauro3]

I'm concerned because the downside of having to pass an instance of an integrator rather then its type is that you would have to generate the integrator outside of the solve function, which burdens the user (the user would have to make sure the integrator and the problem use the same types).

Yes, I second this.

Anyway, maybe we need to first become clear what the type-parameters of the Problem represent. My understanding is that the parameters here: abstract AbstractODEProblem{uType,tType,isinplace} <: DEProblem mean that an instance which is, say, <:AbstractODEProblem{Vector{Float64}, Float64}, means that the algorithm should use Float64 to represent t. Is that correct? Or will we allow to say let a user specify the problem as <:AbstractODEProblem{Vector{Rational{Int}}, Rational{Int}} and then somehow the type can be choosen later, say solve(p, alg, tType=BigFloat). I'm for the first.

[ChrisRackauckas]

I'm concerned because the downside of having to pass an instance of an integrator rather then its type is that you would have to generate the integrator outside of the solve function, which burdens the user (the user would have to make sure the integrator and the problem use the same types).

Yes, I second this.

How is DP5() so much more user-burden than DP5? Only a few will ever have to take (optional) arguments...

Anyway, maybe we need to first become clear what the type-parameters of the Problem represent. My understanding is that the parameters here: abstract AbstractODEProblem{uType,tType,isinplace} <: DEProblem mean that an instance which is, say, <:AbstractODEProblem{Vector{Float64}, Float64}, means that the algorithm should use Float64 to represent t. Is that correct? Or will we allow to say let a user specify the problem as <:AbstractODEProblem{Vector{Rational{Int}}, Rational{Int}} and then somehow the type can be choosen later, say solve(p, alg, tType=BigFloat). I'm for the first.

Yes, the first is what I have implemented. The types will match the problem object, with no override. The one downside to this is that [0;1] is essentially unusable since that makes time integers, so it needs to be clear that it's [0;1.0] for the simplest case Float64. That's why the print makes the types specific.

https://github.com/JuliaDiffEq/DiffEqBase.jl/blob/master/src/problems.jl#L73

I think we should agree to error when types are not supported, instead of doing autoconversions in only same cases.

[mauro3]

Good. I suspect @pwl agrees too, as that is how we do it in PR49. (Regarding the Vector{Int} problem, we could also provide some convenience solve methods which do some auto conversion, e.g. solve(f, tspan, u0, alg=alg)).

Over in PR49 we pass in the alg-type because its type parameters are dependent on, typically, tType, for example here (yep, this code-example is wrong when using units, but the arguments stands that there is some relation between the T in the linked example and tType of Problem). So, our preference would be to pass the type into solve. If not we'll have to add a layer of dummy types which map these dummy instances to the proper algo types (or re-construct alg instance if its type parameters are not correct). This wouldn't be the end of the world if in exchange you get a tangible advantage for passing in instances. (But I don't understand that advantage yet, I'll need to dig through your code a bit more. Any pointers where to look are welcome).

[ChrisRackauckas]

solve(f, tspan, u0, alg=alg))

That can easily ambiguous between problem types. For IMEX, solve(f,g, tspan, u0, alg=alg))? That already conflicts with SDEs.

Over in PR49 we pass in the alg-type because its type parameters are dependent on, typically, tType, for example here (yep, this code-example is wrong when using units, but the arguments stands that there is some relation between the T in the linked example and tType of Problem). So, our preference would be to pass the type into solve. If not we'll have to add a layer of dummy types which map these dummy instances to the proper algo types (or re-construct alg instance if its type parameters are not correct). This wouldn't be the end of the world if in exchange you get a tangible advantage for passing in instances. (But I don't understand that advantage yet, I'll need to dig through your code a bit more. Any pointers where to look are welcome).

I was thinking that CrankNicholson(linersolver=pyamg()) would be a good way to pass algorithm-specific arguments, but I guess linersolver could just be a kwarg and the initialization can handle this. So the difference isn't too much. Let's go with uninitialized alg types? If this is settled I'll just change that right now.

[ChrisRackauckas]

yep, this code-example is wrong when using units, but the arguments stands that there is some relation between the T in the linked example and tType of Problem

The bigger issue there is that you can't lufact or \ matrices with units. There's a way around that but will require a PR to Unitful.

[ChrisRackauckas]

Is that thumbs up another yes on uninitialized alg types?

[pwl]

Is that thumbs up another yes on uninitialized alg types?

Actually for both, I don't think solve(f, tspan, u0, alg=alg)) can work for a wider range of differential problems. EDIT: so the thumbs up where only partial for @mauro3.

The only potential issue might be with passing options to submethods. Are they also going to be passed as kwargs? Then there might be name clashes if there are two submethods both supporting the same keywords.

[mauro3]

Yes, I think all options should just be kwargs. It would be hard to remember which option to pass where.

@pwl: what do you mean with "submethods"?

(And yes, that convenience method may need to be solveode(f, tspan, u0, alg=alg), but that can be discussed later.)

(Where do you deal with those Unitful problems of \ in your code? I'd like to see it implemented.)

[ChrisRackauckas]

(And yes, that convenience method may need to be solveode(f, tspan, u0, alg=alg), but that can be discussed later.)

Is that so much easier than solve(ODEProblem(f,u0,tspan),alg) that it needs to be special cased?

(Where do you deal with those Unitful problems of \ in your code? I'd like to see it implemented.)

I did it for SIUnits, but haven't translated it to Unitful (so ImplicitEuler, Trapezoid, and Rosenbrock methods will currently error on Unitfuls and are shown as disabled in the units_tests.jl). But what you can do for Ax=b is take typeof(b[1]/A[1]), and then call \ on the matrix of .vals, and re-apply units. This requires dispatching on numbers with units and therefore requires the dependency (I'll probably be adding it as a dependency soon, but for other reasons). More difficult is that you'll actually want a pre-allocated unitless arrays for A and b, and pass those in with \. I'll get back to this working again soon.

I still don't know how to handle units in implicit solvers (NLsolve). That's one of many reasons why I said somewhere that I am going to be investigating a lot into Rosenbrock methods (this, and the fact that I can do a lot of pre-calculations in ParameterizedFunctions), and not true implicit methods (BDF/NDF). (I think a very good step for native Julia solvers would be make DASSL a BDF method for ODEs, unless you guys have a BDF method in your back pocket).

[ChrisRackauckas]

@pwl: what do you mean with "submethods"?

I'd assume he means things like tolerances for linear solvers. For example, if you use a cg!, then you might want to set the abstol for cg!, but that conflicts in name with the diffeq solver's abstol. But these issues would be so specific that, while it's good to expose them, they can be the "extra kwargs" for specific packages. This is something that might look a little nicer is passed into the algorithm's constructor, but I think @mauro3 is right that there will still be a documentation issue with finding out what special arguments would go where anyways, so it's probably easier just to document them all as kwargs.

[mauro3]

Ok, here another take of what @pwl means: As somewhat discussed, there should be several levels to the DiffEqBase interface. The top-most level is just that a solver MyAlg<:AbstractAlgorithm needs to implement:

solve(p::Union{All-Problem-Types-it-can-solve}, Alg::Type{MyAlg}; kwargs...) = ...

it could implement its keywords right there, something like:

solve(p::Union{All-Problem-Typs-it-can-solve}, Alg::Type{MyAlg}; 
           abstol=1e-3,
           extra="yay",
           kwargs...) = ... # the extra kwargs... is needed and discard all not need kwargs

How an algorithm then deals with the kwargs is up to them, just pass them on as kwargs or do something more fancy.

However, there could be a deeper layer of the interface which mandates something like the init function I mentioned above. This would then bake the typeof(p), Alg and kwargs together into an instance of the Alg containing all options as fields. Most functionality should be accessible through the upper interface, but it might be nice to have this lower one too, to have some more uniformity with the options.

@ChrisRackauckas this problem with the units: could this not be fixed in SIUnits.jl?

[mauro3]

For sub-solvers, they could just get a dict to themselves. Say solve(..., nlsolve=Dict{Symbol,Any}(:abstol=>1e-3, ...))

[ChrisRackauckas]

@ChrisRackauckas this problem with the units: could this not be fixed in SIUnits.jl?

Yes, that's why I said it needs a PR to Unitful (I changed to Unitful from SIUnits because of the whole ranges/linspace thing, and the fact that SIUnits can't do rational units so it can't do 1/2 time units for SDEs. Supporting was not an option, I forget the reason why but it was pretty clear. Part of it was plotting).

However, there could be a deeper layer of the interface which mandates something like the init function I mentioned above. This would then bake the typeof(p), Alg and kwargs together into an instance of the Alg containing all options as fields. Most functionality should be accessible through the upper interface, but it might be nice to have this lower one too, to have some more uniformity with the options.

So like an abstract type InitializedODE? Each package will probably need a concrete subtype of it, and packages which support it would implement solve(init). If this is what you're talking about, then yes that monster type you found is my initialized type, and I know Sundials' will need to be very different (with the mem), so it makes sense for this deeper level to be a more abstract and package-specific.

BTW I don't know how much performance gains you'll really get out of initializing like this because you'll still have to solve(deepcopy(init)) to solve multiple times to make sure the initialized types will use the same pre-allocated arrays when re-solving, and thus write over each other.

[ChrisRackauckas]

For sub-solvers, they could just get a dict themselves. Say solve(..., nlsolve=Dict{Symbol,Any}(:abstol=>1e-3, ...))

Yeah, this would be something specific packages can do. They'd still have to unpack it for the initialized type in order to not access dictionaries in the inner loop, but this is now getting into details beyond the common interface. This shows that kwargs are powerful enough for whatever is thrown at them though.

[mauro3]

@ChrisRackauckas this problem with the units: could this not be fixed in SIUnits.jl?

Yes, that's why I said it needs a PR to Unitful ...

Ha, thanks for the patience. Sometimes I just take a bit longer...

No, the init-method would just instantiate the Alg type to match the ODEProblem type. alg would also contain the kwargs loaded into properly typed fields. But probably we should not include this in the interface as it adds all the complexity of https://github.com/JuliaDiffEq/Roadmap/issues/5#issuecomment-258814610.

[ChrisRackauckas]

No, the init-method would just instantiate the Alg type to match the ODEProblem type. alg would also contain the kwargs loaded into properly typed fields. But probably we should not include this in the interface as it adds all the complexity of #5 (comment).

And I'm not sure how much you gain from it if it's also not all about pre-allocating the arrays. It seems all of the time is spent in pre-allocating arrays and not option handling. Re-solving with the same pre-allocated arrays would restrict the usage to serial (unless you deepcopy, removing the performance benefit), and if you're solving the same small thing thousands of times you likely are using pmap or @parallel. Do you have a benchmark to show that initing into the alg would decrease the startup time that you'd notice a difference even for small ODEs?

[mauro3]

No benchmarks. I guess my main motivation was to have a unified way of handling the options. But it's too complicated.

[ChrisRackauckas]

I guess my main motivation was to have a unified way of handling the options. But it's too complicated.

I think then that it's an add-on we could play with later.

So we're pretty set on solve? So solve with kwargs as noted before, with alg passed as a type and not an instance of the type.

For the problems, this is what I have slated for DiffEqBase for the ODEProblem and ODETestProblem types:

https://github.com/JuliaDiffEq/DiffEqBase.jl/blob/common_interface/src/problems.jl

Let's get agreement here and then talk about solutions. (Note: in-place is found by >=3 parameters because things like ParameterizedFunctions make extra dispatches (for parameters) which have more arguments).

[mauro3]

So we're pretty set on solve? So solve with kwargs as noted before, with alg passed as a type and not an instance of the type.

Yes, that sounds good to me and works for PR49.

Concerning the Problem:

• Should we call it ExplictODEProblem and also have an ImplicitODEProblem with f(t,u,du,residual). And also one where a mass matrix is specified?
• If we use the same types for ODEs and DAEs then maybe use IVP in above type names?
• Julia convention is to (usually) have the mutated argument first, so f(du,t,u). Would we want to follow this? (I'm not sure)
• Jacobian's etc. are passed through ParameterizedFuntctions.jl, right? I still need to look at this. When does this mean that ParameterizedFunctions.jl become a dependency? For any implicit solvers?
• Is the ODETestProblem needed or could the analytic solution be passed around differently? But irrespective, policy should probably be to dispatch on AbstractODEProblem and let subtypes add extra fields compared to ODEProblem.

[ChrisRackauckas]

Should we call it ExplictODEProblem and also have an ImplicitODEProblem with f(t,u,du,residual). And also one where a mass matrix is specified?

Isn't an ImplicitODEProblem just the general form for a DAE? I think that's easier to understand as DAEProblem, and would propose that for the common interface: DAEProblem(f,u0,du0,tspan) with function signature f(t,u,du,residual) (as the in-place version).

Mass matrices... do you have a good idea? Instead of a "usually blank function" (t,u,du)->nothing and a boolean value type, maybe it's better to have a separate MMODEProblem <: AbstractODEProblem which expects a mass matrix? What's good shorthand we should use (I'll write it as mm for now)? mass_matrix? m?

I think we should allow it to be parametric typed: mm::M and let M subtype Any. The reason is because in many cases it might make sense for M to be some kind of operator type, with the canonical being I the UniformScaling operator, but I could see matrix-free Laplacians and the like being useful. Since usually it's used in computing something like M - lambda*J, these operator types would work directly, so it would be up to the algorithms as to whether they support them. And if it's parametric typed, then a type-check:

if mm<: Base.Callable
  mm(t,u,M) # Set the values of the mass matrix
else
   M = mm
end
W[:] .= M-lambda*J

will work in all cases and the branching will be pre-computed at compile time.

Julia convention is to (usually) have the mutated argument first, so f(du,t,u). Would we want to follow this? (I'm not sure)

I did this at first for this reason (as f(du,u,t)), but switched everything to f(t,u,du) because that's the common C/Fortran signature. Otherwise you have to put a closure over everything that does interop which isn't the end of the world, but is unnecessary.

Jacobian's etc. are passed through ParameterizedFuntctions.jl, right? I still need to look at this. When does this mean that ParameterizedFunctions.jl become a dependency? For any implicit solvers?

No, not everything would need ParameterizedFunctions.jl. In fact, nothing would. This is a bigger point so let me handle that separately.

Is the ODETestProblem needed or could the analytic solution be passed around differently? But irrespective, policy should probably be to dispatch on AbstractODEProblem and let subtypes add extra fields compared to ODEProblem.

Yes, I just have solve dispatch on AbstractODEProblem and at the very end do a type-check for if to calculate errors (this branch compiles away, so for non-test problems it's non-existent). You could also have this dispatch for a different solution type which includes the analytical solution and errors. In general we should say that the interface we agreed on is for AbstractODEProblem and ODEProblem is just one instantiation of an AbstractODEProblem which satisfies the interface. There may be cases where someone wants to build a different AbstractODEProblem, but still have the solvers "just work". I actually have an example of this in FinancialModels.jl where you can make things like a HestonProblem which satisfies the (earlier version which will be updated to match our ODE changes) AbstractSDEProblem interface, and so the SDE solvers work directly on it. This is nice because you can store the extra information about your problem in the type, but have it still solve correctly if f, u0, and tspan exist in there (etc., basically if it at least satisfies the interface then the solvers work on it).

[pwl]

So we're pretty set on solve? So solve with kwargs as noted before, with alg passed as a type and not an instance of the type.

All looks good to me!

Should we call it ExplictODEProblem and also have an ImplicitODEProblem with f(t,u,du,residual). And also one where a mass matrix is specified?

Couldn't we differentiate between these problems using isinplace?

typealias ExplicitODEProblem ODEProblem{uType,tType,2}
typealias InPlaceODEProblem ODEProblem{uType,tType,3}
typealias ImplicitODEProblem ODEProblem{uType,tType,4}

In that case it would be better to have isinplace as a symbol, maybe something in the lines of [:explicit,:inplace,:implicit].

If we use the same types for ODEs and DAEs then maybe use IVP in above type names?

I agree, there should be a root abstract type for general IVPs, with uType and tType as parameters.

Julia convention is to (usually) have the mutated argument first, so f(du,t,u). Would we want to follow this? (I'm not sure)

This is the way to go.

Also, since it is a new package, and a base file for the whole organization, @ChrisRackauckas would you consider switching it to 4 indent? If you look at Julia base all the files there are indent 4. I know this is not your style but 2 indent could cause a headache for other contributors.

Jacobian's etc. are passed through ParameterizedFuntctions.jl, right? I still need to look at this. When does this mean that ParameterizedFunctions.jl become a dependency? For any implicit solvers?

ParametrizedFunctions are a subtype of Function, and Base.Callable=Union{DataType,Function} so this looks general enough.

On a different note, wouldn't having a Union type as a type field introduce type instability?

Is the ODETestProblem needed or could the analytic solution be passed around differently? But irrespective, policy should probably be to dispatch on AbstractODEProblem and let subtypes add extra fields compared to ODEProblem.

Yeah, I don't see why ODETestProblem should belong to base. Either move the analytic to ODEProblem or move ODETestProblem somewhere else (DiffEqDevTools.jl maybe?).

[ChrisRackauckas]

On a different note, wouldn't having a Union type as a type field introduce type instability?

Function is already an abstract type in v0.5. Making this strict would make your solvers recompile for every function given to it. You can already do that if you want with a function barrier, but I suspect that this causes more problems than gains.

Yeah, I don't see why ODETestProblem should belong to base. Either move the analytic to ODEProblem or move ODETestProblem somewhere else (DiffEqDevTools.jl maybe?).

Sure, it can move to DiffEqDevTools.jl.

Also, since it is a new package, and a base file for the whole organization, @ChrisRackauckas would you consider switching it to 4 indent? If you look at Julia base all the files there are indent 4. I know this is not your style but 2 indent could cause a headache for other contributors.

Yeah... this is just the headache of Juno defaulting to 2 indent. I honestly don't care, I just never changed the default. I'll get to that later, probably just find a tool to convert the whole thing.

Couldn't we differentiate between these problems using isinplace?

Implicit ODE Problems will need to hold a du0 for a lot of problems, and a separate type for du0 (rateType is what I use) in order to be units compatible. Why not make them seprate subtypes of AbstractODEProblem? If your algorithm can handle them all, you can just dispatch on the highest class.

[pwl]

Function is already an abstract type in v0.5. Making this strict would make your solvers recompile for every function given to it. You can already do that if you want with a function barrier, but I suspect that this causes more problems than gains.

Right, this makes sense.

I did this at first for this reason (as f(du,u,t)), but switched everything to f(t,u,du) because that's the common C/Fortran signature. Otherwise you have to put a closure over everything that does interop which isn't the end of the world, but is unnecessary.

I wouldn't be too upset about keeping f(t,u,du) but consider this. While f(du,u,t) makes implementing the Fortran interface inconvenient, it could be a one time inconvenience (depending on how much maintenance you expect) but then you would be stuck with a nonstandard signature when implementing pure Julia packages.

Yeah... this is just the headache of Juno defaulting to 2 indent. I honestly don't care, I just never changed the default. I'll get to that later, probably just find a tool to convert the whole thing.

Thanks! I wonder, why did they make it the default for Juno then?

Implicit ODE Problems will need to hold a du0 for a lot of problems, and a separate type for du0 (rateType is what I use) in order to be units compatible. Why not make them seprate subtypes of AbstractODEProblem? If your algorithm can handle them all, you can just dispatch on the highest class.

True, I forgot about that. In PR49 we handled it in a different way, we had a concrete type IVP, which held the fields common for the other problems we implemented (explicit and implicit ones) and we used parameters to specialize this general type as e.g. an ExplicitODEProblem (see https://github.com/JuliaODE/ODE.jl/blob/master/src/base.jl#L51). The IVP we designed was general enough for our purposes, but this approach might not be general enough for what we are working on here. I wouldn't mind having more flexibility at the expense of a bigger abstract type hierarchy and possible code duplications between various IVP types, so I'm ok with what you implemented of ODE Problems.

[mauro3]

Function is already an abstract type in v0.5. Making this strict would make your solvers recompile for every function given to it. You can already do that if you want with a function barrier, but I suspect that this causes more problems than gains.

Right, this makes sense.

I'd be happier if no function barrier was needed and the ODEProblem could be passed around as a whole. So, it would be:

type ODEProblem{uType,tType,isinplace, F<:Base.Callable} <: AbstractODEProblem{uType,tType,isinplace}
  f::F
  u0::uType
  tspan::Vector{tType}
end

Thanks! I wonder, why did they make it the default for Juno then?

That is Mike Innes preference. I guess no-one is that bothered either way thus there was no outcry.

[pwl]

I'd be happier if no function barrier was needed and the ODEProblem could be passed around as a whole.

The downside is an increased number of type parameters, but it I think it's worth it.

[ChrisRackauckas]

If you do

type ODEProblem{uType,tType,isinplace, F<:Base.Callable} <: AbstractODEProblem{uType,tType,isinplace}
  f::F
  u0::uType
  tspan::Vector{tType}
end

Then there's no such thing as "precompiling the ODE solvers", since you would have to re-compile the solvers for every new function (in Julia since v0.5, every function and anonymous function is its own type. They are just overloaded types with in-accessible type names). I am not sure that forcing the lack of precompilation is a good thing, and as I said, an algorithm can always opt in to act like this with a function barrier (I probably won't be doing it, because for things like the 14th order Feagin method the precompilation time is pretty long!).

[mauro3]

But for performance the function implementing the algorithm will need to be compiled specifically for each f. And it's clear that this cannot be done in precompilation as f is not known. Or what am I missing?

[ChrisRackauckas]

Here's the full deal with ParameterizedFunctions. Let me first explain what they are in more detail, and then give what I'm thinking for their relation to ODE solvers.

Disconnect the macro from the ParameterizedFunction. In fact, ParameterizedFunctions.jl mixes two things:

1. The ParameterizedFunction interface
2. A macro for making a full parameterized function using symbolic tools (SymEngine.jl)

The ParameterizedFunction type is in DiffEqBase, so you could just make a type which is a ParameterizedFunction without using ParameterizedFuntctions.jl. The specification is just about call overloading: it's just an extensive interface over call-overloaded type. Here's most of the interface specification (which keeps growing):

f.a # or f[:a], accesses the parameter a
f.jac_exists # Checks for the existence of the explicit Jacobian function
f(t,u,du) # Call the function
f(t,u,du,params) # Call the function to calculate with parameters params (vector)
f(t,u,2.0,du,:a) # Call the explicit parameter function with a=2.0
f(t,u,2.0,df,:a,:Deriv) # Call the explicit parameter derivative function with a=2.0
f(t,u,J,params,:param_Jac) # Call the explicit parameter Jacobian function
f(t,u,J,:Jac) # Call the explicit Jacobian function
f(t,u,iJ,:InvJac) # Call the explicit Inverse Jacobian function
f(t,u,H,:Hes) # Call the explicit Hessian function
f(t,u,iH,:InvHes) # Call the explicit Inverse Hessian function

Right now there's booleans for declaring which of the functions exist (though that will be changing to be parameters, matching what we did with ODEProblem since that lets everything dispatch smoother). (The other details is that the Symbol dispatch is just syntactic sugar over a value-type symbol dispatch. Maybe the spec should just get rid of it because it's unnecessary. Note that these value-type dispatches compile away in functions when the symbol is known ahead of time, so if you write f(t,u,2.0,du,:a) it compiles all of the extra stuff away, and so this just becomes a way of saying which dispatch to use).

So the ParameterizedFunction is a huge extensive (and growing) interface for an overloaded type, and booleans to declare which functions exist. The macro then exists because I would think that in most cases no one would want to declare a lot of this by hand (example, each parameter gradient), while having these pre-computed leads to performance gains when they exist. This interface is extensible enough for everything from bifurcation plots to parameter estimation, and holding most (but not all, which is why it's still growing) of the functions which are used for speeding up calculations in all of these settings.

---

The full ParameterizedFunction is much more than what's needed here. What we could have is a DiffEqFunction interface which is much more constrained. For example, we could do something as simple as

abstract DiffEqFunction <: Function

Then define the interface that the function is f(t,u,du) with Jacobian f(t,u,J,Val{:Jac}). To define it, you'd just make and overload a type:

immutable MyFunctionType <: DiffEqFunction end
f = MyFunctionType()
function (f::MyFunctionType)(t,u,du)
  # Declare the in-place function to calculate du
end
function (f::MyFunctionType)(t,u,J,Val{:Jac})
  # Declare the in-place function to calculate J
end

# Now f(t,u,du) and f(t,u,J,Val{:Jac}) work

The downside is that it will require users to use call-overloading, and this syntax is pretty new (though I don't think it's hard to understand once you see it). The upside is that the function can have as many extra details as someone wants (all the way to a ParameterizedFunction), and none of the other interfaces are cluttered (otherwise you'd need a field in the problem for the Jacobian which in many cases will be "empty", or use a duplicate problem types for ones that add Jacobians and all the extra things).

Or we could have DiffEqFunction be a type with value parameters for what kinds of parameters exist and have FunctionWithJacobian be an alias for DiffEqFunction which only has Jacobians. This call-overloading stuff is very flexible and allows someone to add details to the API which others can ignore (for example, I could in my algorithms type-check in some rootfinders for explicit Hessian functions, but this kind of stuff never has to show up in the ODEProblem interface and could just be a footnote in my own personal docs like "if you make an overloaded function which also includes this in the signature, then the ____ solver will use it"). Otherwise you need "a spot" for everything you want to include. Maybe the Jacobian is common enough to give it a spot (and using dispatch on these you can declare the Jacobian by a closure jac = (t,u,J) -> f(t,u,J,Val{:Jac})).

[mauro3]

Thanks for this explanation. I will not have time to ingest it until tonight and will comment then.

[ChrisRackauckas]

But for performance the function implementing the algorithm will need to be compiled specifically for each f. And it's clear that this cannot be done in precompilation as f is not known. Or what am I missing?

Normally you don't have to compile functions specifically to other functions to get performance. The only way this helps is if you decrease the call latency by allowing it to inline. If you pass f to g and g compiles for f, you only get performance improvements if it can eliminate the function call. This is not always possible (tends to only happen for very simple functions), can maybe be forced with @inline, and still doesn't give that much of a performance increase (it gives a performance increase if it's a cheap function you are calling very often, like looping through and using + requires that + inlines to get good performance, but I don't think this kind of usage is common in differential equations). The performance advantage is dependent on the cost of the function call (which is somehow proportional to the number of arguments?) versus the time to compute the function itself (if it's a non-trivial function, you'll never notice the difference).

I could benchmark this to see how much it really matters (using the function barrier, I would just have to change my ODEIntegrator internal type you found to strictly type the function). I would suspect that the gains would be very minimal if even measurable, while the precompilation times can be very costly in many cases (leading to a bad user experience)

[mauro3]

This is not what my mental model is but I might be wrong. Here a quick test of this: https://gist.github.com/mauro3/c15870dbde6766f4f04b94169b9d78ef

Notes:

• the untyped field leads of type Ato inferred Any
• the typed field of type B leads to the same code as passing in the function directly
• but performance is worse for the typed field

Maybe these contradictory results are because of the tiny function used (sin)?

[ChrisRackauckas]

the untyped field leads of type A to inferred Any

Declarations fix this.

Maybe these contradictory results are because of the tiny function used (sin)?

Most likely. I'm trying it on actual DEs right now to see.

[ChrisRackauckas]

On the 100x100 linear ODE, I measure the time difference as being pretty stable around 3e-4 to 4e-4 seconds across tolerances:

abstols = 1./10.^(3:13)
reltols = 1./10.^(0:10);

So at the lowest tolerances here this gives a 15% speedup, and at the lowest tolerances a 0.6% speedup. Since this is a a still cheap but not insignificant costing function call (linear, but 100x100 matrix is looped through so it's not like something simple likex^2), it seems to show you can get a good speedup on smaller repeated calls at the cost of precompilation. Is that worth it?

Note that I did say:

Declarations fix this.

You have to make the field access type-inferrable, and you may need to help it happen. While it's not hard to do if you're used to it, very simple uses like the one you showed are performance traps. Maybe the argument to compile to each function is less about performance and more about safety. Also, I realized that you can function barrier the other way as well (for loose inner functions) so it makes almost no difference as to which is chosen for the problem type.

[pwl]

I don't know if we are still considering a common Options type, but if that's the case there are core functions like norm or isoutofdomain (we it in PR49) that benefit a lot from inlining, even for problems with complicated equations. This has no direct influence on the top-level interface, but if we are going for a common Options type we should use typed fields in it, and if we do that precompilation would be impossible anyway. Although common options type may simply land in DiffEqDevTools and exclude precompilation only for packages that explicitly use DiffEqDevTools.

In any case, I personally think compile times are not so critical. Speaking from my experience (which represents only a small part of use cases), the problems I was dealing with took much longer to solve then to compile. Precompilation would only benefit problems that are quick to solve but long to initialize, but I don't know how popular these kind of problems are. But again, this is my personal, and maybe limited, use case.

[ChrisRackauckas]

I don't know if we are still considering a common Options type, but if that's the case there are core functions like norm or isoutofdomain (we it in PR49) that benefit a lot from inlining, even for problems with complicated equations.

I don't think we are still considering a common Options type, though in the interface internalnorm is considered (not named norm because that can conflict with Julia's norm functions). isoutofdomain wasn't part of the interface, though I'd argue you'd always want to do this with event handling (you'd want the actual time and the step where you went out of the domain, not where the algorithm happened to be and out of the domain). But yes, these will inline even from kwargs if you @inline them.

This has no direct influence on the top-level interface, but if we are going for a common Options type we should use typed fields in it, and if we do that precompilation would be impossible anyway. Although common options type may simply land in DiffEqDevTools and exclude precompilation only for packages that explicitly use DiffEqDevTools.

Not necessarily. If the functions were ::Function and these were strict, you can have a precompilation which works for any problem on Vector{Float64} and Float64, with the DEFAULT_NORM. But yes, if the norm is changed then it will recompile, and this could help it inline.

In any case, I personally think compile times are not so critical. Speaking from my experience (which represents only a small part of use cases), the problems I was dealing with took much longer to solve then to compile. Precompilation would only benefit problems that are quick to solve but long to initialize, but I don't know how popular these kind of problems are. But again, this is my personal, and maybe limited, use case.

Yeah, that's probably the case. I just hate the fact that complicated methods like the 14th order Feagin take like 30-40 seconds to compile, but this keeps getting better as Julia gets improved. But as you say, for cases like that the compilation cost is probably still negligible (if you're using such a high order method, then you're solving at very low like 1e-40 precision and it will take some time anyways)

[pwl]

complicated methods like the 14th order Feagin take like 30-40 seconds to compile

wow, that's some long compile time. I can only imagine how long the test suite takes to complete.

[pwl]

One more tiny thing in the Problem type. If the tspan is always going to be Vector{tType} of length 2 (is it?), shouldn't it be a Tuple{tType,tType} instead?

[ChrisRackauckas]

wow, that's some long compile time. I can only imagine how long the test suite takes to complete.

elapsed time: 393.18273478 seconds locally. Travis takes much longer. But that's the price you have to pay if you want to make sure each feature of each algorithm works. The tests catch just about anything.

One more tiny thing in the Problem type. If the tspan is always going to be Vector{tType} of length 2 (is it?), shouldn't it be a Tuple{tType,tType} instead?

This is very much a micro-optimization and I would be surprised if it ever truly made a performance difference. However, it's far more restrictive. I think I may have forgotten to mention one aspect of tspan. I currently interpret tspan as "step to every value in the interval" since there are cases where you may want to constrain the stepping (i.e. known discontinuity or location of stiffness) or need to save at specific values and don't have an interpolation available (SDEs) (this is a cheap-man's way of mixing adaptivity with the Vector of dt @mauro3 wants). I recommend this be part of the standard interface since most people really want values that are like the saveat-style (i.e. saving at specific times using interpolations) but in some cases that's only possible through this slightly more constrained interface (again, adaptive timestepping in stochastic equations)

When we get the type-system upgrade, I think we could handle tspans as a combination: tspan::T where

T<:Union{NTuple{N,tType},AbstractVector{tType}},N,tType

but that clearly is using a lot of triangular dispatch.

[ChrisRackauckas]

I also want to note that the feature I just mentioned is super easy to implement. Instead of making it a while t<T loop, you just do

@inbounds for T in Ts
    while t < T

where Ts = tspan[2:end] (or Ts = @view tspan[2:end]). It's probably similarly easy with iterators.

[pwl]

My comment was more about being restrictive and specifying the standard then about optimization. I agree that this is going to be a popular and important use case but it could be easily and more transparently handled with keywords.

I keep thinking of an ODEProblem as a representation of a Cauchy problem, and that's why I'm against including technical or algorithm related stuff in ODEProblem. I agree that t1 could be put in there, as it makes sense mathematically (we often consider Cauchy problems on a finite time interval, even the direction of time matters for their solvability) it makes little sense to me to add the intermediate steps there. Still, some problems might call for something like these intermediate steps, especially the discontinuities you mentioned. In that case interpreting tspan as a set of "special" point for the equation could be the way to go. But if we know that these are special points from the mathematical point of view, we shouldn't mix them with the ordinary endpoints [t0,t1] and we should give them a separate type field instead (tdiscontinuous or maybe even a separate problem type ODEDiscontinuousProblem?).

That said, I would rather not have them be used for output points, step points or stiff points (although these last ones might be loosely interpreted as a feature of a given Cauchy problem). We have plenty of keyword arguments we could use: tout, tstiff, tsteps and so on.

[ChrisRackauckas]

I keep thinking of an ODEProblem as a representation of a Cauchy problem, and that's why I'm against including technical or algorithm related stuff in ODEProblem. I agree that t1 could be put in there, as it makes sense mathematically (we often consider Cauchy problems on a finite time interval, even the direction of time matters for their solvability) it makes little sense to me to add the intermediate steps there. Still, some problems might call for something like these intermediate steps, especially the discontinuities you mentioned. In that case interpreting tspan as a set of "special" point for the equation could be the way to go. But if we know that these are special points from the mathematical point of view, we shouldn't mix them with the ordinary endpoints [t0,t1] and we should give them a separate type field instead (tdiscontinuous or maybe even a separate problem type ODEDiscontinuousProblem?).

That is a good point. And a tsteps argument makes sense. I find this very reasonable. I'll wait on @mauro3's opinion. Also, should save_timeseries be renamed to save_every_tstep?

Where were we on the DAE problems? I am pretty sure we at least need DAEProblem (name pending) for the implicit ODE functions since those require a du0, and so at that point we have a whole new problem. However, for mass matrices, there were two threads: separate problem type vs optional mass matrix and a boolean value type as a type parameter. I was for the separate problem type. Where was everyone else?

[ChrisRackauckas]

Starting the discussion on Solutions, I propose the following interface:

Basic Interface

The required fields are:

u::uType
t::tType
dense::Bool

The following interface is put on the DESolution type which all solutions derive from:

Base.length(sol::DESolution) = length(sol.u)
Base.endof(sol::DESolution) = length(sol)
Base.getindex(sol::DESolution,i::Int) = sol.u[i]
Base.getindex(sol::DESolution,i::Int,I::Int...) = sol.u[i][I...]
Base.getindex(sol::DESolution,::Colon) = sol.u

Dense Interface

Additionally, if dense=true, like in lots of ODE solutions, there is a field interp which holds a function interp(t) which calculates the interpolation at every point in t (t can be a vector, this allow for many optimizations). In this case, the following interface is used:

(sol::DESolution)(t) = sol.interp(t)

Test Interface

Lastly, there is a test interface which requires the fields

u_analytic
errors::Dict{Symbol,uEltype}

for holding an analytical solution and a dictionary of errors. You can have many or as few errors in the dictionary, and in DiffEqDevTools I put it into 3 levels:

Standard: :final is the L1 error at the final timestep Timeseries Errors: l2 and l∞, i.e. the l errors at the saved points. This is only meaningful if save_timeseries=true. Dense Errors: L2 and L∞, i.e. take 100 points uniformly spaced in the timespan and use the interpolations to get the errors at each of these points. This requires dense output.

What this looks like to a user

What this looks like to a user is:

sol[i] # Gets the ith value of u, the saved timeseries
sol.t[i] # Gets the timepoint of the ith save
sol(t) # Gets the interpolated solution at `t`

I believe (and have been told by others) that this is very intuitive and makes it so that way you don't need to any of the behind-the-scenes implementation details.

What's also nice is that any tool can be written on this interface and it will work for the solution from any algorithm. For example, check out the AbstractODESolution plot recipe included in DiffEqBase:

https://github.com/JuliaDiffEq/DiffEqBase.jl/blob/common_interface/src/plotrecipes.jl

This makes it so that plot(sol) will plot the solution of each component (using the dense output to smooth the plot if available). Someone can add an option for phase plots and things like that, and it will "just work".

Another case where the common solution interface is nice is because all of the components are written simply on the common interface. So DiffEqParamEstim.jl performs parameter estimation on DEProblems by only needing the dense solution interface (this could be relaxed with tstop changes). Global sensitivity analysis could do something similar. So with this you can write tools which work independent of the problem/DE solving algorithm. That ends up being super powerful!

ODEs

For ODEs, I propose that everything subtypes AbstractODESolution <: DESolution. Algorithms/packages may have to have difference concrete solution types. One case I already know of (and have mentioned) is Sundials, which will need a special mem field in order to implement the interpolation interface.

Other Options

One thing that can be added is two booleans to DESolution: DESolution{Dense,Analytic} where those are boolean value types for whether those extra parts of the interface are implemented. Then dense doesn't have to be a field. And we would then make it AbstractODESolution{Dense,Analytic} as well.

[ChrisRackauckas]

"Common" Solution

However, even though there will be special cases, I propose a "common" solution. I know that this one is general enough to hold OrdinaryDiffEq, PR49, and ODE solutions with their interpolations:

type ODESolution{uType,uEltype,tType,rateType,P,A} <: AbstractODESolution
  u::uType
  u_analytic
  errors::Dict{Symbol,uEltype}
  t::tType
  k::rateType
  prob::P
  alg::A
  interp::Function
  dense::Bool
end

Again, we can have an ODETestSolution separate from ODESolution if you like. The constructor I use does two fancy things. First, it builds the interp function in it:

  dense = length(k)>1
  saveat_idxs = find((x)->x∈saveat,t)
  t_nosaveat = view(t,symdiff(1:length(t),saveat_idxs))
  u_nosaveat = view(u,symdiff(1:length(t),saveat_idxs))
  if dense # dense
    if !isinplace && typeof(u[1])<:AbstractArray
      f! = (t,u,du) -> (du[:] = prob.f(t,u))
    else
      f! = prob.f
    end
    interp = (tvals) -> ode_interpolation(tvals,t_nosaveat,u_nosaveat,k,alg,f!)
  else
    interp = (tvals) -> nothing
  end

The fancy part is that the saveat points don't have a full k stack (which for Hermite interpolation is just the derivative, for other interpolations this is a Vector of Arrays). The only thing that needs to be defined is ode_interpolation(tvals,t_nosaveat,u_nosaveat,k,alg,f!), i.e. how do you interpolate at tvals knowing the values at t_nosaveat are u_nosaveat and k, and the function you solved was f! (needed for lazy adding steps to k) with alg.

I then calculate all of the possible errors:

  errors = Dict{Symbol,eltype(u[1])}()
  if !isempty(u_analytic)
    errors[:final] = mean(abs.(u[end]-u_analytic[end]))

    if save_timeseries && timeseries_errors
      errors[:l∞] = maximum(vecvecapply((x)->abs.(x),u-u_analytic))
      errors[:l2] = sqrt(mean(vecvecapply((x)->float(x).^2,u-u_analytic)))
      if dense && dense_errors
        densetimes = collect(linspace(t[1],t[end],100))
        interp_u = interp(densetimes)
        interp_analytic = [prob.analytic(t,u[1]) for t in densetimes]
        errors[:L∞] = maximum(vecvecapply((x)->abs.(x),interp_u-interp_analytic))
        errors[:L2] = sqrt(mean(vecvecapply((x)->float(x).^2,interp_u-interp_analytic)))
      end
    end
  end

A boolean is there for "don't calculate too much" since the dense errors can be costly.

Of course, you could reject this instantiation of the interface for something different (I don't know how you setup your interpolations, I just put a Hermite on yours before using your derivative outputs).

[ChrisRackauckas]

As pointed out by a user, it would make a lot of sense to add one more interface to the solution object: an iterator interface.

https://github.com/JuliaDiffEq/DifferentialEquations.jl/issues/108

This way [somefun(s) for s in sol] works. Since it's on the solution object and not on the Problem or solve, it's cheap (i.e. it doesn't solve the differential equation again, just uses the computed values) and has some compelling use cases in analysis (though as mentioned in that issue, replacing sol with sol.timeseries (or after the proposed changes, sol.u) would work).

The code to implement this could just go into DiffEqBase:

function start(sol::DESolution)
  #sol.tslocation = state
  1
end

function next(sol::DESolution,state)
  state += 1
  #sol.tslocation = state
  (sol,state)
end

function done(sol::DESolution,state)
  state >= length(sol)
end

function eltype(sol::DESolution) 
  eltype(sol.u)
end

Now the other thing that could be involved is this tslocation field. If that field exists, then this iterator interface automatically makes animate(sol) work from Plots.jl, meaning that the solutions will be trivial to animate (at each step they will just use the plot recipe, and tslocation is part of the plot recipe). This makes animating really easy which is really nice for time-dependent PDEs (I have already set this all up with FEMSolution to animate solutions to the heat equation). Thus I would suggest adding this to the standard solution interface or make there be a value parameter on DESolution for whether this field exists.

[mauro3]

I haven't caught up with all yet but I had a read of and thought on the ParameterizedFunctions. In a way it's pretty radical. Instead of specifying an interface in the ordinary sense: a type needs to implement this and that generic function, it goes the other way and says: this generic function unifies all these related but somewhat disjoint methods. Taking this to its full conclusion (I'm not advocating that we should) is that all methods needed to solve a DE could be lumped into f, say also the norm f(x, Val{:norm}), or a nonlinear solver, or what not, and called that way. As you can see, I've only started to digest this.

One thing which occurred to me, which may be from my only very cursory look at ParameterizedFunctions, is that you seem to go through hoops to determine which methods are defined. Could instead method_exists be used (with some care):

julia> f(::Val{1}, x) = 1
f (generic function with 1 method)

julia> f(::Val{2}, x) = 1
f (generic function with 2 methods)

julia> f{T}(::Val{3}, x::T) = 1
f (generic function with 3 methods)

julia> methods(f, Tuple{Val{1},Vararg})
# 1 method for generic function "f":
f(::Val{1}, x) at REPL[15]:1

julia> methods(f, Tuple{Val{2},Vararg})
# 1 method for generic function "f":
f(::Val{2}, x) at REPL[16]:1

julia> methods(f, Tuple{Val{3},Vararg})
# 1 method for generic function "f":
f{T}(::Val{3}, x::T) at REPL[17]:1

[ChrisRackauckas]

One thing which occurred to me, which may be from my only very cursory look at ParameterizedFunctions, is that you seem to go through hoops to determine which methods are defined. Could instead method_exists be used (with some care):

Or have type parameters. But yeah, it's just checking for dispatches. Maybe using method checks like that is easier and works at compile time. If so, that would cut back on the interface quite a bit.

But yeah, that's the general idea. It allows you to add a bunch of things to f. Since it can bake in the parameters, it makes total sense for ParameterizedFunctions and makes parameter estimation and the like trivial to implement. For "balls to the wall" optimizations like "pre-computed in-place Rosenbrock W-inverse functions", this allows a clean way for many users to ignore it, but for it to exist. So I think that in those two cases it's clear. It CAN also treat Jacobians. However, Jacobians are common, so the question is then should this be the way to treat Jacobians in the problem type, or should Jacobians be "special-cased" and get their own field in the problem type?

[ChrisRackauckas]

OrdinaryDiffEq.jl is compliant with the things we have discussed:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_solve.jl#L1

I have the interface ready for Sundials (now doesn't use the simplified API and instead directly uses Sundials, which will allow it to extend to the full common interface later) and will make a PR when this is all settled:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_solve.jl#L271

I setup a dispatch for ODEInterface and started discussions with the author:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_solve.jl#L189

https://github.com/luchr/ODEInterface.jl/issues/9

I have starter code for PR49:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_solve.jl#L448

[mauro3]

That is a good point. And a tsteps argument makes sense. I find this very reasonable. I'll wait on @mauro3's opinion. Also, should save_timeseries be renamed to save_every_tstep?

Yes to make tspan a tuple and then add one or several kwargs to do specify output points, dense output, etc.

Concerning typed vs untyped function fields in ODEProblem. Here an updated gist: https://gist.github.com/mauro3/c15870dbde6766f4f04b94169b9d78ef, which I think mimics a "workload" of a solver reasonably closely. It shows:

• fully typing the field puts it on par with passing the function directly
• adding type annotations does help but does not bring it on par with the typed field
• using a function barrier puts it on par
• inlined vs not-inlined does not change this
• (in the first gist I showed I forgot a const, thus the strange results)

This suggests to me that we should type the function field. Or is this test-gist flawed?

[ChrisRackauckas]

This suggests to me that we should type the function field. Or is this test-gist flawed?

Nope, this is what I expected. Let's go with strict typing of the function fields. Of course, the performance gains are a lot less in "most" diffeq applications because of the time not spent in function evaluation (so for things like Adams and BDF methods you see almost no difference, and I noted that you can see like a 15% difference in RK methods). However,

using a function barrier puts it on par adding type annotations does help but does not bring it on par with the typed field

The case you're missing is putting a const and a declaration, and pulling the function out. Of course, at this point you can see that, while there are ways around it, I fear that newcomers to the ecosystem could easily fall into performance traps if they attempt to naively program a method, and so we should strictly type the functions to avoid this. I already made the change to DiffEqBase (I had it sitting on a feature branch).

Yes to make tspan a tuple and then add one or several kwargs to do specify output points, dense output, etc.

Okay, let's go with that. The saveat kwarg already covers points for dense output. So we can just add tstops kwarg to cover non-interpolation timestep constraints.

[mauro3]

I'm not quite sure what you mean with below statement, could you post an example:

The case you're missing is putting a const and a declaration, and pulling the function out.

For the functions which have super long compile times, would there be a trick to let them precompile but not specialize. Probably involving ANY.