Run @pwl's PR #49

[mauro3]

It be great to have a branch which runs with PR https://github.com/JuliaLang/ODE.jl/pull/49 to test its performance. @obiajulu, fancy having a go at this?

[obiajulu]

Definitely, this sounds like a great idea. I'm still investigating what is currently breaking with ob/j04-fix, and think I should have that sorted out soon enough. How does waiting to merge ob/j04-fix into master, then making this new branch sound?

[mauro3]

Yep, that sounds good. Although remove the reference to your solver, so it runs on stock ODE.jl.

[obiajulu]

Currently, the travis build is still failing on ob/j04-fix, but only on nightly. The build passes for v0.4. I think the problem is with the Requires.jl package. Others have seemed to stumble into the same issues as recently as two days ago (see the following issue MikeInnes/Requires.jl/issues/24 and an associated travis build with similar error). I think it is safe to assume it is not on our end.

[pwl]

Calling a solver without initstep keyword is not equivalent to calling the solver with initistep=NaN. In the first case the dtinit is triggered, in the second it is not. Setting initstep=NaN shouldn't be the way to resort to the default keyword values. I'm inclined to say that calling a solver with any of the arguments (including the keyword ones) set to NaN should result in an error. We even have a test function for that isoutofdomain.

Another thing is that it would be nice to have an actual call to the solver printed out or to return some callable object, so one can trace edge cases like like this without going through the source code of IVPTestSuite.

[pwl]

Can we take a look at why there is a discrepancy between the tests in ODE.jl and IVPTestSuite for fixed step RK and ode23s? I believe if we manage to run IVPTestSuite for this PR we should merge it as dev.

[pwl]

I'm getting -1 significant digits on the plei test case for fixed step RK solvers:

julia> runtestsuite(ODEsolverfns=[ODE.ode1])

Running test case plei with solver ODE.ode1
Running test 1 of 6: sig. digits= -1.3967199479995205, walltime= 1.815471144s, memory= 117.0MB
Running test 2 of 6: sig. digits= -1.3967199479995205, walltime= 1.997339832s, memory= 184.0MB
Running test 3 of 6: sig. digits= -1.3967199479995205, walltime= 3.737129655s, memory= 853.0MB
[...]

and whatever that means for ode23s

julia> runtestsuite(ODEsolverfns=[ODE.ode23s])

Running test case bruss1d with solver ODE.ode23s
Running test 1 of 6:Error occurred: MethodError(convert,(Float64,Dual(0.007708008643947041,-10038.00237712215,1.0125802781986795,5020.02,0.0,0.0,0.0,0.0,0.0,0.0,0.0)))
 sig. digits= NaN, walltime= NaNs, memory= -1bytes
Running test 2 of 6:Error occurred: MethodError(convert,(Float64,Dual(0.007708008643947041,-10038.00237712215,1.0125802781986795,5020.02,0.0,0.0,0.0,0.0,0.0,0.0,0.0)))
 sig. digits= NaN, walltime= NaNs, memory= -1bytes
Running test 3 of 6:Error occurred: MethodError(convert,(Float64,Dual(0.007708008643947041,-10038.00237712215,1.0125802781986795,5020.02,0.0,0.0,0.0,0.0,0.0,0.0,0.0)))
 sig. digits= NaN, walltime= NaNs, memory= -1bytes
Running test 4 of 6:Error occurred: MethodError(convert,(Float64,Dual(0.007708008643947041,-10038.00237712215,1.0125802781986795,5020.02,0.0,0.0,0.0,0.0,0.0,0.0,0.0)))
 sig. digits= NaN, walltime= NaNs, memory= -1bytes
Running test 5 of 6:Error occurred: MethodError(convert,(Float64,Dual(0.007708008643947041,-10038.00237712215,1.0125802781986795,5020.02,0.0,0.0,0.0,0.0,0.0,0.0,0.0)))
 sig. digits= NaN, walltime= NaNs, memory= -1bytes
Running test 6 of 6:Error occurred: MethodError(convert,(Float64,Dual(0.007708008643947041,-10038.00237712215,1.0125802781986795,5020.02,0.0,0.0,0.0,0.0,0.0,0.0,0.0)))
 sig. digits= NaN, walltime= NaNs, memory= -1bytes

This looks like something having to do with Jacobian (because of the Dual from ForwardDiff showing up), but I run some tests and the ForwardDiff jacobian gives exactly the same results as an explicit Jacobian (I also added tests for this kind of mismatch to ODE.jl). Can it be a problem internal to IVPTestSuite?

[mauro3]

The -1 significant digits means that it is way off. But it seems to converge to something as the digits stay the same.

I'll look into the second issue. Does it work with the FD jacobian?

[mauro3]

The fixed step solver might have problems with the test cases though. Have you seen https://github.com/mauro3/IVPTestSuite.jl/blob/master/results/results.md, this is what it should look like +/-.

[mauro3]

I don't get the same error with ode23s. But I get one too, enabling the full backtrack:

Running test case bruss1d with solver ODE.ode23s
ERROR: InexactError()
 in setindex! at array.jl:314
 in fn! at /home/mauro/.julia/v0.4/IVPTestSuite/src/testcases/bruss1d.jl:61
 in fn at /home/mauro/.julia/v0.4/IVPTestSuite/src/solvers/solvers.jl:13
 in anonymous at /home/mauro/.julia/v0.4/ODE/src/helpers.jl:74
 in _calc_jacobian at /home/mauro/.julia/v0.4/ForwardDiff/src/api/jacobian.jl:101
 in jacobian at /home/mauro/.julia/v0.4/ForwardDiff/src/api/jacobian.jl:27
 in anonymous at /home/mauro/.julia/v0.4/ODE/src/helpers.jl:74
 in jac! at /home/mauro/.julia/v0.4/ODE/src/types.jl:43
 in start at /home/mauro/.julia/v0.4/ODE/src/ode23s.jl:81
 in start at /home/mauro/.julia/v0.4/ODE/src/dense.jl:44
 in map at ./essentials.jl:153
 in collect at array.jl:261
 in collect at /home/mauro/.julia/v0.4/ODE/src/types.jl:226
 in ode at /home/mauro/.julia/v0.4/ODE/src/interfaces.jl:32
 in ode_conv at /home/mauro/.julia/v0.4/ODE/src/interfaces.jl:84
 in ode23s at /home/mauro/.julia/v0.4/ODE/src/interfaces.jl:60
 in ODEjl_wrapper at /home/mauro/.julia/v0.4/IVPTestSuite/src/solvers/ODE.jl:36
 [inlined code] from util.jl:210
 in run_ode_test at /home/mauro/.julia/v0.4/IVPTestSuite/src/running_tests.jl:12
 in run_ode_testsuite at /home/mauro/.julia/v0.4/IVPTestSuite/src/running_tests.jl:49
 in runsuite_ODEadaptive at /home/mauro/.julia/v0.4/IVPTestSuite/testsuites/suite_ODE_adaptive.jl:7
 in runtestsuite at /home/mauro/.julia/v0.4/IVPTestSuite/testsuites/runsuites.jl:96

[mauro3]

For some reason a ForwardDiff number gets fed to the objective function:

Running test case bruss1d with solver ODE.ode23s
map(typeof,(t,y,dydt)) = (Float64,Array{Float64,1},Array{Float64,1})
map(typeof,(t,y,dydt)) = (Float64,Array{Float64,1},Array{Float64,1})
map(typeof,(t,y,dydt)) = (Float64,Array{Float64,1},Array{Float64,1})
map(typeof,(t,y,dydt)) = (Float64,Array{Float64,1},Array{Float64,1})
map(typeof,(t,y,dydt)) = (Float64,Array{ForwardDiff.GradientNumber{1000,Float64,Array{Float64,1}},1},Array{Float64,1})

(Printed by adding @show map(typeof,(t,y,dydt)) here: https://github.com/mauro3/IVPTestSuite.jl/blob/6f4b940ab2089f5e1dda8c8650ffbd628540775e/src/testcases/bruss1d.jl#L49)

In that file, the test-case is defined. Maybe you can set it up to run it directly?

[obiajulu]

Also, I just added plotting functionality to the terminalmode branch. If you run plottestsuite(), PyPlot plots of the most recent test suite run will be plotted. I thought it might come in handy when debugging.

[pwl]

How do you enable the backtrack?

[mauro3]

Move this statement out of the try-block: https://github.com/mauro3/IVPTestSuite.jl/blob/6f4b940ab2089f5e1dda8c8650ffbd628540775e/src/running_tests.jl#L13

(Sorry, IVPTestSuites.jl is not the best piece of software...)

[pwl]

Isn't brusselator supposed to use it's own Jacobian?

[pwl]

By the way, the keyword for the Jacobian is now jac!=..., so ODE is not catching the Jacobian you are passing from IVPTestSuite. But it doesn't explain the Duals in the objective function.

[mauro3]

I just remembered: Here is the bruss taken apart, with a call to ode23s: https://github.com/mauro3/ODE.jl/blob/64b63dc3d896ca575c3ace0c4964c33a7c1bc5af/examples/bruss1d.jl Probably best to use that.

[mauro3]

Ok, the problem is with the in-place function and its wrapping. Not sure what a general approach would be. See: https://gist.github.com/mauro3/d8a127ab798d8655c3151f3a21a6fa85

It works with the old FD jacobian.

[pwl]

Ok, so the good news is that all fixed step methods from ODE.jl converge (for a simple oscillator problem): https://gist.github.com/pwl/de8594a0384a1906d9a596045db89836. The problem seems to be either a difficult test case or a bug in IVPTestSuite.

[pwl]

I think I found the issue. You are not passing the initstep argument to fixed step solvers: https://github.com/mauro3/IVPTestSuite.jl/blob/master/src/solvers/ODE.jl#L29. In ODE.jl the step size is not determined from tspan and you have to explicitly use initstep to decrease the step size.