Performance benchmark vs. Python / Mathematica / Matlab / etc.?

[lancejpollard]

What do you think about the idea of documenting some benchmarks comparing mathjs with some of these other math libraries? I don't know much about any of them, but it would awesome to see how they compare in some simple calculations.

[josdejong]

Yes that will be interesting.

I think the performance will be quite terrible right now, we haven't focused yet on high performance matrix calculations. That will be addressed after version 1.0 is reached. There are a couple of interesting possibilities to improve the performance like using typed arrays, flat instead of nested arrays, etc.

See related discussions at #59 and #71.

[lancejpollard]

Cool, good to know. Looking forward to it!

[lancejpollard]

@josdejong have you started on anything for https://github.com/mathjs? It would be awesome if they were micro components.

[josdejong]

I'm indeed planning to split mathjs into a set of separate components and move them to https://github.com/mathjs. Haven't actually started with that, but already registered the organization name.

[philippsimon]

I made several performance tests comparing Matlab, some JavaScript libraries (Math.js, jsfeat, scijs), NumPy and Julia. I adapted for that the performance tests from Julia: https://github.com/JuliaLang/julia/tree/master/test/perf/micro I think it would be nice if there would be project that compares the performance of the different (javascript) math libraries.

[philippsimon]

Here are benchmarks of several JavaScript libraries: https://github.com/mikolalysenko/ndarray-experiments

[josdejong]

Great, thanks Philipp, good benchmarks give us the opportunity to improve and tweak the performance in a focused way. There is a lot of room for performance improvements in math.js :)

I can't find any math.js tests though, can you point me to them?

[philippsimon]

I just pushed my changes to the Julia tests here: https://github.com/philippsimon/julia/blob/master/test/perf/micro/perf.js

Here my results:

javascript,fib,0.141
javascript,parse_int,0.797
javascript,mandel,0.547
javascript,quicksort,1.06
javascript,pi_sum,42
javascript,rand_mat_stat,22.5
javascript,rand_mat_mul,3283
javascript,rand_mat_mul_ndarray,3402
javascript,rand_mat_mul_mathjs,59691
# I think the values for Julia are: min, max, mean, std
julia,fib,0.133796,66.814950,0.144926,0.570599
julia,mandel,0.252195,3.062956,0.265681,0.084233
julia,quicksort,0.417816,12.685040,0.463544,0.273521
julia,pi_sum,33.556671,43.695805,34.816103,2.310412
julia,rand_mat_stat,27.057084,111.874389,41.443800,16.317529
julia,rand_mat_mul,112.676486,195.935792,123.447862,19.883687
python,fib,0.0
python,parse_int,0.0
python,mandel,0.0
python,quicksort,14.9998664856
python,pi_sum,636.000156403
python,rand_mat_stat,196.000099182
python,rand_mat_mul,139.999866486

I have more tests but I didn't do them with Math.js as the performance was not interesting. I mainly looked more at the source code of Math.js as the math parser is really interesting and I checked how difficult it would be to replace the Math.js matrix implementation with ndarray as they are much faster but sadly quite ugly to use.

[josdejong]

Thanks.

We still have plans to replace the implementation of the Matrix class using nested arrays with flat arrays. This would a allow using of typed arrays, and allows to improve the performance of quite some matrix operations. I'm not sure if we could use ndarray for this, or use it for inspiration or as a base: we also want to have support for other than numeric values (i.e. complex, bignumber, fraction, ...).

Lots of room for improvement I think :)

[philippsimon]

So for complex matrices they already have an implementation: http://scijs.net/packages/#scijs/ndarray-complex

For bignumber I didn't see any module.

From in 2 months on I could help you with the project. I just think it would probably reduce the work if the matrix implementations are used from ndarray, as several developers work already on it and it seems to be already quite stable.

[josdejong]

Thanks for your offer Philipp! If we could built upon ndarray would be great, that saves a lot of work. We will have to investigate if that's feasible and if so in what form.

[josdejong]

Closing this discussion now because it has been inactive for a long time. Feel free to reopen if needed.