Open ivan-pi opened 9 months ago
Yes, right now that benchmark has been not ideal since we have not had a MATLAB license to be able to update it. So it's currently published with all of the information as to what is ran on what computers for full transparency, knowing that we have a somewhat non-ideal scenario here. If anyone has a license and can run that benchmark we would be really happy to accept a PR with a locally ran version of the benchmark that improves that.
Describe the bug 🐞
The results of the Simple Interval Rootfinding benchmark claim a 33x speed-up compared to the MATLAB solver, however the conditions of the benchmark were not the same.
The main differences are:
Expected behavior
The benchmarks should be "optimal, fair, and reproducible". This one fails in the "fair" comparison category.
When I run the MATLAB code on an Intel i7-11700K @3.60GHz (Rocketlake), the elapsed time is ~0.12 s or roughly the same as using the NonlinearSolve.jl bisection method. (Caveat - the hardware is different.)
Additional context
The benchmark was presented at JuliaCon 2023 (see https://youtu.be/O-2F8fBuRRg?si=GF24GyZEBek0Yi-Y&t=1022) with a claim of an additional 5-fold speed-up or a time of under 0.01 s. An un-merged pull request is mentioned in the video, but I was not able to locate it.