drym-org / qi

An embeddable flow-oriented language.
59 stars 12 forks source link

Baseline performance audit #20

Open countvajhula opened 2 years ago

countvajhula commented 2 years ago

The current performance benchmarks in profile/ that are run via make profile are inaccurate, and additionally, are not comprehensive.

  1. The existing benchmarks need auditing to ensure that the results are in accord with actual performance, with the benchmarking apparatus properly factored out.

E.g. @michaelballantyne reports:

the difference between the flat and recursive benchmarks is an artifact of your benchmarking infrastructure. In particular, check-value constructs a list that is as long as the number of iterations of the test, as take is an operation over lists rather than sequences:

(for ([i (take how-many (cycle inputs))]) (fn i))

It looks like the cost of constructing that list accounts for much of the time in the "Conditionals" benchmark, for example. With a testing setup that doesn't construct such a list, the Racket implementation is much faster than Qi. I suspect that in the recursive benchmarks the cost of the computation is higher relative to the construction of the list of inputs and therefore is more visible in the benchmarks.

  1. In addition, benchmark coverage is sparse at the moment, and should be made more comprehensive -- e.g. probably expand the benchmarks in profile/forms.rkt to include all Qi forms (currently these include only a few such as relay and group).
countvajhula commented 2 years ago

Note for down the line: it would be great to add continuous benchmarking into the CI workflow, so that the performance effects of each commit are automatically tracked:

https://github.com/benchmark-action/github-action-benchmark

This will likely involve calling the existing benchmarking functions in profile/* but generating the output in the JSON format described at the above link, i.e. {"name": <name-of-the-benchmark>, "unit": "ms", "value": <the-measured-value>}.

These should be runnable via a Makefile target, e.g. make report-benchmarks, which can be run in the GitHub Actions workflow just like the existing targets for testing, etc.