search

amazon-ion / ion-c

A C implementation of Amazon Ion.
https://amazon-ion.github.io/ion-docs/
Apache License 2.0
166 stars 43 forks source link

Add performance regression testing workflow #332

Closed nirosys closed 1 year ago

nirosys commented 1 year ago

Issue #, if available: n/a

Description of changes: This PR adds a GHA workflow that builds both the HEAD of the target branch, as well as the PR's revision, and runs the ion-bench tool to generate benchmark results for a set of 200KiB binary ion datasets.

This PR adds only the read (deserialize_all) benchmark to the workflow. An update to add the write (serialize_all) benchmark will follow.

The workflow uses google-benchmark's compare.py which performs a null-hypothesis check agains the data to determine whether the results from both benchmark runs are statistically similar or not. This test is currently not used for the validation. This is primarily because I need to understand the tuning of the alpha parameter, as well as what to expect from the test in a noisy environment like GHA.

Example Run: here

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

linlin-s commented 1 year ago

Would you mind adding the sample run of this workflow to the description?

nirosys commented 1 year ago

Would you mind adding the sample run of this workflow to the description?

@linlin-s, I'm not sure what you mean. The workflow ran as part of the checks, so all of the output is available there. The full output is a bit much for the PR description, do you mean just the output of the compare.py?

linlin-s commented 1 year ago

Would you mind adding the sample run of this workflow to the description?

@linlin-s, I'm not sure what you mean. The workflow ran as part of the checks, so all of the output is available there. The full output is a bit much for the PR description, do you mean just the output of the compare.py?

Apologize for the confusion. What I mean by sample run is already included in the checks. We can track the current workflow run in the checks since it is triggered by pull_request to master. After we update the trigger event, we might have to manually trigger the workflow to see the results since the workflow will be triggered only when the source code changed. In this case it would be easier to review if we could include the hyperlink of the updated workflow run.