search

SciML / DiffEqDevTools.jl

Benchmarking, testing, and development tools for differential equations and scientific machine learning (SciML)
https://benchmarks.sciml.ai/
Other
48 stars 35 forks source link

Save all errors in `WorkPrecision` instead of just the one passed as `error_estimate`? #125

Closed nathanaelbosch closed 11 months ago

nathanaelbosch commented 12 months ago

I often find myself running a benchmark, but then I'm often interested in not just one metric but many (https://github.com/SciML/DiffEqDevTools.jl/pull/124 comes from a similar motivation, as I'm not always only interested in visualizing runtime, but also the step size or the number of vector-field evaluations). What I could do right now is to run multiple WorkPrecisionSet with different error_estimate arguments, and then plotting each of these. But depending on the experiment, that takes a lot of time. Why not just save all available errors into the WorkPrecision object?

If we were to add this, we could still pass an error_estimate argument and save it into both WorkPrecision and WorkPrecisionSet to keep the current behaviour. But additionally, we could add new plotting recipes that enable plotting other errors by specifying some keyword argument. (My personal favorite would actually be some functionality similar to idxs in the standard ODE plots, such that we could flexibly plot certain metrics against each other, even things like time vs nf, or errors against each-other; but that could be a separate issue once the information is saved in the objects).

What do you think? I locally have a minimal working example that I could turn into a PR.

ChrisRackauckas commented 12 months ago

That makes sense. Though the API should take an array of error estimate definitions, so the user can provide a different list. The reason for this is that the SDE case should only default to the strong error estimates, since the weak error estimates can be an order of magnitude (or more) longer to actually compute.

nathanaelbosch commented 12 months ago

I'm very unfamiliar with all the parts that are not just ODEs, so I'm not sure I completely get your point. But overall my thought was that it would make sense to save all the errors that are already computed, i.e. to not just consider errsol.errors[error_estimate], but all the keys that are in that dict. I wouldn't compute any additional errors, so this change should not come with an increased compute time I believe?

ChrisRackauckas commented 11 months ago

I see what you mean, yes save everything that is actually computed. In some cases there are flags to compute more, and I'm just saying those should still be off by default.

nathanaelbosch commented 11 months ago

Closed with #126