More benchmarks

[flashultra]

There are some benchmarks on https://benchmarksgame-team.pages.debian.net/benchmarksgame/ which give nice comparison between different languages. The benchmarks are pidigits, fannkuch-redux, fasta, reverse-complement , spectral-norm, binary-trees , k-nucleotide, n-body, mandelbrot and regex-redux with the source code for the corresponding language. Maybe some of them could be implemented on Haxe ? The other comparison could be target compiled from Haxe source code with the target from native code. At example reverse-complement test compile from java code against Haxe java ( jvm) export . Both resutl would be jar files.

[AlexHaxe]

You've probably seen that we already have binary-trees, nbody and mandelbrot benchmarks. And we've been gradually adding more benchmarks. So "More benchmarks" is underway, however there is no clear definition of "done" in your issue. So I would ask you to open individual issues for benchmarks you find relelvant. If you can provide Haxe code that would be even better. And I definitely won't say no if you want to attempt a pull request for new benchmarks... :)

For your second suggestion: running native code binaries as benchmarks to our benchmarks is an interesting idea (I think someone on Discord came up with it a while back). We would have to accomodate these additional results, but I guess we wouldn't need to run them all the time. We can generate results once (even manually) and then mix them into benchmark pages where available. The question would then be to find native code, set it up the same way our Haxe code version works and then record run times (I don't think compile times are relevant here, but even those could be added). Again we should track this in a separate issue.

[flashultra]

Yes, I just mention what benchmarks are available on benchmarksgame-team.pages . Thank you.

[flashultra]

I wonder about two more questions here:

1) How correct are results from Haxe benchmarks ? For example here : https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/rust-gpp.html we have for n-body ( C++) test is running for 2.14 secs and bench Haxe show for c++ under 1 second. Is it Haxe implementation 2x faster ?

2) Is it possible to see mem consumation in the benchmark?

[AlexHaxe]

I'm not sure (and I haven't compared sources) if Haxe's n-body benchmark uses the exact same parameters as the Rust one. I don't remember if we had to e.g. reduce the number of iterations in order to make sure more target platforms would be able to complete the benchmark.

in general the benchmarks are meant to demonstrate how the same Haxe code performs on different targets. they are not built to be comparable / compatible with other benchmark collections. additionally those benchmarks exist to track performance improvements or degradations over time with different Haxe versions.

currently there is no measurement of memory consumption. if you know of an easy way to do it accross all targets, I'm sure it can be added.

[flashultra]

My idea was to compare against other benchmarks ( as the one mention above ) , so could give rough estimation about native vs Haxe compiled code ( C++ vs hxcpp for example).

About memory, maybe there is no easy solution. Openfl use System.totalMemory to get the memory currently in use.

Some other benchmarks , such as Programming-Language-Benchmark , have a separate tools ( write in C# - Program.cs and ProcessUtils.cs which run the CI actions and measure the results.

I'm not sure what solution will be more applicable for Haxe - to check memory in xx msec and get the maxmemory or to run program in other language ( C# ) and get values.

[AlexHaxe]

https://programming-language-benchmarks.vercel.app/haxe-vs-rust already has a section for Haxe generated code and it allows comparison between Haxe and other languages. since their system seems to be set up the way you propose, I don't think we need to copy that effort. it would be more beneficial to add to their project by writing Haxe ports of benchmarks, making sure they use the latest versions, maybe they allow multiple Haxe versions, etc. (I don't know how active that project is)

[flashultra]

I think you are right. About measuring memory consumtion in Haxe I think there a two easy options : 1) Using this bash script: memusg COMMAND [ARGS]... , which should looks like this in haxe : memusg haxe run.hxml and parsing the results. 2) Using similiar Python script : memusg python . Example : memusg [OPTS] haxe run.hxml