Add utility for benchmarking vs ndarray NumPy (Addresses #18)

[peterlionelnewman]

In response to: create a utility to benchmark vs ndarray, NumPy #18

Was pretty confusing -> I have no idea how the ha-ndarray crate works and if I'm implementing the right functions or hardware acceleration at all!

[peterlionelnewman]

Thanks Haydn,

I'm removed the DS_Store file. The ha_ndarray_test module can be generated in your python environment by building the benchmark folder with maturin, i.e. (pip install maturin, then maturin develop --release).

from pypi "maturin develop builds the crate and installs it as a python module directly in the current virtualenv. Note that while maturin develop is faster, it doesn't support all the feature that running pip install after maturin build supports."

Given I'm not sure I'm implementing opencl properly, it would be worth running the benchmark after building on your machine to see if the graphs differ significantly.

Cheers,

[haydnv]

OK that makes sense now, thanks! When I try to run the benchmark I get these errors: a

$ maturin develop --release
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at .../ha-ndarray/benchmark/venv/bin/python
    Finished release [optimized] target(s) in 0.02s
⚠️  Warning: Couldn't find the symbol `PyInit_ha_ndarray_comparison` in the native library. Python will fail to import this module. If you're using pyo3, check that `#[pymodule]` uses `ha_ndarray_comparison` as module name
📦 Built wheel for CPython 3.11 to /tmp/.tmpEp7Djy/ha_ndarray_comparison-0.1.0-cp311-cp311-linux_x86_64.whl
🛠 Installed ha_ndarray_comparison-0.1.0

$ python3 benchmark.py 
Traceback (most recent call last):
  File ".../ha-ndarray/benchmark/benchmark.py", line 12, in <module>
    from ha_ndarray import ndarray_test
ModuleNotFoundError: No module named 'ha_ndarray'

Also, I think you should add fig.png to the .gitignore file and not include it in the PR since it's generated by the code you're submitting.

[peterlionelnewman]

Hi Haydn, apologies I’m away from the computer for a while on holiday between jobs. When I’m back I’ll make this work the other way (run python from rust, rather than running rust from python) much should be simpler for the setup and environment. Unfortunately this won’t be till December. Til then

[haydnv]

Thanks Pete! I also need a graphical benchmarking utility for other projects so I'll take some time next month to figure out why maturin is not working as expected for me.

[peterlionelnewman]

Hey Haydn, made this commit which removes the python pyo3 approach and instead runs python from shell with the simpler 'python -c "script here"' and then parses the python output from that to get the python timings.

the timing of the 3 crates/modules are then printed in a table for comparison

I think for 'let size_max: u32 = 8;' this works though I might have an error in 'fn max_value()' which is still making over/underflow errors for large arrays?

Let me know how you go!

[haydnv]

Stray observations:

• please update the README to describe the new code (e.g. the command to run the benchmarks, etc)
• The timings for Python are so much faster for many operations, and so uncorrelated to data size, that I wonder if there's some optimization at work like not computing a result that's not read
• please remove fig.png

Would you consider adapting this code to run using criterion? This would provide a graphing utility for free and probably make the code easier to maintain. You've already got the basic structure of the individual benchmarks in place.

[peterlionelnewman]

Thanks haydn,

I'm made the requested updates and simplified some of the array generators.

I've also included in the benchmark readme an example output of the benchmarking script with the #s I get which seem to be power law increasing corresponding with the power law increase in the size of arrays.

I've added the todo for the addition of criterion in the future!