Benchmark suite

[nikomatsakis]

We need to have a start at a standard Rayon benchmark suite. @cuviper got us started with https://github.com/nikomatsakis/rayon/pull/32. But we need a bit more.

• [x] quicksort -- everybody's favorite :)
• [x] mergesort or heapsort?
• [x] NBody -- I've started porting NBody from the ParallelJS benchmarks in the nbody branch.
• [x] Mandelbrot -- @willi-kappler has implemented Mandelbrot, we could freeze a version (I'd want to have something constant for comparisons).
• [ ] ProceduralTerrain from the ParallelJS benchmarks
• [ ] liquid-resize from the ParallelJS benchmarks
• [ ] A gigantic dot-product implemented using par_iter
• [x] Naive matrix multiplication implemented using par_iter perhaps?
• [ ] Raytracer (perhaps based on https://github.com/matklad/rustraytracer?)
• [ ] Audo codec, perhaps?

More suggestions or pointers welcome.

We also need some kind of script to compile and run the demos and collect the data. Ideally, we'd run it regularly on some server so we can track performance automatically.

[willi-kappler]

I think mandel-rust v0.4 should be ready for this (will be released soon). I plan to do some clean up and add script-collectable output. Should I remove the other crates and just leave Rayon for the benchmark suite?

Matklad wrote a raytracer in Rust which could be added. Maybe also audio en-/decoder (ogg, flac) and compression / decompression algorithms. IIRC I saw some of them written in 100% Rust.

There should always be a way to verify (automatically) that the results are correct (compare images, compressed files, etc.).

[nikomatsakis]

@willi-kappler

I think mandel-rust v0.4 should be ready for this (will be released soon). I plan to do some clean up and add script-collectable output. Should I remove the other crates and just leave Rayon for the benchmark suite?

If you could prepare a kind of stripped down version to include as a benchmark/demo, that'd be great.

Matklad wrote a raytracer in Rust which could be added. Maybe also audio en-/decoder (ogg, flac) and compression / decompression algorithms. IIRC I saw some of them written in 100% Rust.

Good ideas. The raytracer should be imminently actionable.

There should always be a way to verify (automatically) that the results are correct (compare images, compressed files, etc.).

Yes. I need to add this to the nbody demo that I wrote, though I'm not quite sure how to do it. I mean I guess I can just hardcode some results. (I was considering running the original JS code for some time and comparing the results, as well.)

[lilianmoraru]

What I propose is for it to be data driven. For example, to have JSON config files that indicate which tests to run, how many times, how to collect the script output(so you if you add another field at the output for example, you don't need to adapt the script but add it in that config file). The script could expect some specific JSON file names at some specific locations that can overwrite some of the defaults(of course, the defaults are also in a predefined location). It could have a hard requirement on jq and use it to parse the config files. The script results could also probably be displayed in the JSON format so it can parsed by other tools or scripts.

For this to be started(I could do this but after a week), as a starting point, the output from the benchmarks should be predefined.

[willi-kappler]

@lilianmoraru That sounds great! The benchmarks should at least have the following output:

• time taken (in ms ?)
• result validation (true / false) (so the result should be validated inside the benchmark. All the data structures + functions are there. Alternatively the script could compare the result of each benchmark but this will make the script more complicated.).
• options used ?

This output should be written to a file, probably also JSON. Maybe s.th. like this: {"benchmark": {"timeTaken": 1234.56, "valid": true, "options": "--iter 1000 --no_img"}}

[cuviper]

I'm pretty happy with the results of the prime sieve I just wrote -- want it as a demo? https://users.rust-lang.org/t/how-to-make-this-parallel-prime-sieve-faster/4802/26

[willi-kappler]

I've just read it on the rust user forum and was going to suggest it here ;-)

So: sure, that would be really nice, thanks!

I've been too busy the last couple of weeks with other stuff, but will finish up my mandelbrot demo for the benchmark suite.

[nikomatsakis]

@cuviper sounds good; and we should do another release so that par_chunks_mut is available

[lilianmoraru]

@nikomatsakis So do you think that would be a good way to go with script + jq(if the JSON configs is even the way to go)? Or may be instead of jq to have it written in Rust? My worry would be that jq is not available with every distro(or MSYS) or it is to old, that would mean that the script should download the dependency(site might be down), but it cannot rely on being able to install jq(user rights, MSYS?)...

Also, what's your opinion on moving neg-tests-*, pos-tests-* in a test(s) folder?

[nikomatsakis]

I'd be inclined to try and make something pure-rust, myself. Maybe we can just make a combined "demos" crate that can run all the benchmarks.

[nikomatsakis]

The main advantage of something pure rust is that it doesn't add any "extra" deps (we can manage everything through cargo) and hence will be trivial to run.

[lilianmoraru]

@nikomatsakis And what are your thoughts on the configuration and behavior to be defined in JSON config files? Or would you prefer it to be directly written in Rust? What I was thinking: you can give the modified configuration file to somebody and "here, run with this".