Usage with non-Python projects

[SylvainCorlay]

I am interested in airspeed velocity for continuously benchmarking a pure C++ project

What would you recommend to do so?

• use the current Python API to call a compiled executable eventually?

Or is there a specified achema for the JSON assets to be generated that I could try to generate from a native benchmarking solution?

[pv]

Most future-proof is probably to use track_* benchmarks and your own timing routines, either calling dynamically linked library or executable. The compilation/build issue is not currently solved however, only setup.py is supported.

There's a somewhat informal asset schema is here: https://asv.readthedocs.io/en/stable/dev.html#benchmark-suite-layout (replace "stable" with "latest" to see master branch).

[SylvainCorlay]

Thanks. Now experimenting. Thankfully we have been using conda for all our projects, and you support conda environments instead of only python virtual environments, so we already have a clear path for the dependencies.

[SylvainCorlay]

So the setup.py seems to be called all the time.

We could either

• use a shim setup.py that does the build
• maybe provide a hook for build / installation of the package.

[pv]

699 should help with the custom build commmand part

[egpbos]

I'm also experimenting with this and the custom build commands work nicely.

As a simple first benchmark I'd like to just run my entire program with a certain configuration. I was thinking of just using a time_ function that runs the program using subprocess.run. Would there be an advantage to using track_*?

[danielsoutar]

I'm also looking into using asv with the project I'm working on.

I've noticed that if you have the JSON data in the results directory, an empty set of install/uninstall/rebuild commands and the .asv-machine.json file set, then there is no need for the project to be Python at all - and this JSON data could be from anywhere, provided it's rearranged into the kind of format that asv expects.

Is this a sensible way of using asv? I'm aware it's not quite the original intention, but the web visualisation component is really valuable as a general-purpose stand-alone tool and my team are not wanting to migrate from using Google Benchmark on a pure C++ project, with the C++ codebase situation being similar to @SylvainCorlay .

[pv]

For future-proofness I would recommend the following: have a "wrapper" benchmark suite written in Python, which:

1. Runs the actual benchmarks in setup_cache, using whatever benchmark suite you are using, and saves the results to a temporary file in some format.
2. Has a benchmark generator (metaclass, factory function etc.) that generates a track_* methods with pretty_name etc. attributes set, each of which simply extracts the obtained benchmark results saved in the temporary file. Or, if your benchmark suite supports running benchmarks one by one, you can call it to run them directly here instead of using setup_cache.

ASV's json file format is documented to some degree, but there's no strong guarantee of backwards compatibility.

[SylvainCorlay]

I'm also looking into using asv with the project I'm working on. [..]

@danielsoutar thanks for the heads up!

ASV's json file format is documented to some degree, but there's no strong guarantee of backwards compatibility.

@pv would you consider instead creating a formal spec for that file format?

[pv]

I don't want to fix the file format at this stage. Basically, there are likely still several changes needed, and not constraining development outside the main use cases is better at this stage.

On May 19, 2019 9:59:03 AM UTC, Sylvain Corlay notifications@github.com wrote:

I'm also looking into using asv with the project I'm working on. [..]

@danielsoutar thanks for the heads up!

ASV's json file format is documented to some degree, but there's no strong guarantee of backwards compatibility.

@pv would you consider instead creating a formal spec for that file format?

[saraedum]

I am using a somewhat hacky way to track Google C++ benchmarks with asv at https://github.com/flatsurf/conda-recipes/blob/master/asv/__init__.py. It uses a metaclass approach to generate something that ASV can understand. It's not perfect but might be of interest to people trying to do something similar.

[mforets]

Interesting! In my case we'd like to track a Julia project with asv. Still have to figure out how it works, but the comments in this thread are certainly a good starting point.