I would love for benchmark functions to be able to return a value. This can be used for several things:
User can test benchmark functions
Verify that the returned value is consistent for each implementation variant
If specified, perhaps the output of the benchmark functions can be given
It is good practice for the user to provide a reference implementation of the function in the benchmark file. For example, if toolz.groupby is being benchmarked, this is as simple as doing from toolz import groupby in the benchmark file. Another option is to define it in the benchmark file, such as def groupby(key, seq). The reference implementation isn't benchmarked, but it does allow benchmark functions to be tested.
If no reference implementation is given, the output of each variant can still be compared to make sure they are all equivalent. If there is a reference implementation, then it is possible to identify which variants are incorrect.
I would love for benchmark functions to be able to return a value. This can be used for several things:
It is good practice for the user to provide a reference implementation of the function in the benchmark file. For example, if
toolz.groupbyis being benchmarked, this is as simple as doingfrom toolz import groupbyin the benchmark file. Another option is to define it in the benchmark file, such asdef groupby(key, seq). The reference implementation isn't benchmarked, but it does allow benchmark functions to be tested.If no reference implementation is given, the output of each variant can still be compared to make sure they are all equivalent. If there is a reference implementation, then it is possible to identify which variants are incorrect.