Benchmarking suite - requirements?

[kwohlfahrt]

I'm planning to add a profiling module, to make it easier to see how changes to the core affect the performance. I'd like some feedback about what kind of transforms, wavelet lengths and image dimensions (size, dimensionality, power-of-two-ness, etc) are commonly used.

I'm processing 2D and 3D signals with dwtn and swtn, db1-4 wavelets, about 100-300 pixels a side.

Now that #158 is open, I'm making a list here of more concrete bits that need doing.

• [x] Separate benchmarks for C code
• [x] Benchmarks for (i)swt(n)
• [x] Documentation
• [ ] Host benchmark results somewhere?

[rgommers]

Do you mean profiling, as in https://github.com/rkern/line_profiler? Or benchmarks? Your description suggests the latter. If so, an ASV benchmark suite would be really nice to have. See https://pv.github.io/scipy-bench/ for how that looks for scipy. Such benchmarks are really easy to write, so doing lots of image sizes shouldn't be a problem. I'd expect that sizes relevant for image processing aren't limited to powers of two.

[kwohlfahrt]

Yes, benchmarks is the right word. I've changed the title. ASV looks good, I'll dig into that.

[kwohlfahrt]

I've tried setting up asv, but it is unable to build the package. It complains that ImportError: No module named numpy.distutils.core. It is available on my system, so this is related to the virtual environment asv tries to set up.

The normal way of letting pip know is by adding install_requires and setup_requires to the setup function arguments. However, it complains (Unknown distribution option: 'install_requires'), but only when running ./setup.py build, not ./setup.py egg_info. I assume numpy.distutils does not support this argument and I haven't been able to find any other way of specifying dependencies.

So, unless there is a way of doing this with numpy's distutils, the first step will be to move away from those and back to vanilla setuptools. Thoughts?

[rgommers]

Can you share your branch? asv works for Scipy, and numpy needs to be found by asv anyway to run the benchmarks.

[rgommers]

However, it complains (Unknown distribution option: 'install_requires'), but only when running ./setup.py build, not ./setup.py egg_info. I assume numpy.distutils does not support this argument

The warning comes from distutils.

Maybe we should update to setuptools and the right form of install_requires. But I'd really really like to avoid setup_requires - pip doesn't support that so this triggers the horrible mess that is easy_install.

[kwohlfahrt]

@rgommers sure, it's the asv branch. The first commit is unrelated, and the second is pretty much just the result of creating the 'asv' directory and running asv quickstart. Fails on my work machine and my laptop.

[kwohlfahrt]

I'm not a pip/distutils expert by any means, so if you know what the modern way of doing things is that's great.

[kwohlfahrt]

It can be worked around by specifying the dependencies to ASV rather than in setup.py. I pulled out the last of the templating and converted the setup over to vanilla setuptools (with the exception of tests) before I figured that out though, so PRs for both incoming.

[rgommers]

@kwohlfahrt sorry for the inaction - I've been very busy at work and hit by the flu epidemic here. Will do some reviewing now.