Splits the workflow into parallel steps by python version and sample data. Within each parallel job, it tests both read and write for the sample data using the baseline and the tip of the PR.
Adds a step to cache PIP packages to help reduce the setup time in the parallelized jobs. Each parallel job runs in <10 minutes, so we don't need to wait for ages to see the results.
Moves configurable parameters that we might want to tweak into environment variables at the start of the file.
Removes the unnecessary PR-content-check workflow
This builds on #282 and must be merged after it. The perf workflow will not run successfully until this PR is rebased on top of #282. If you prefer, I can combine this into a single PR, but I thought it might be easier to review if I break them up.
You can see an example of the new workflow here.
There is still some work to be done on getting the right metric for the performance benchmark comparison. I'm beginning to suspect that ops/s might be more stable for slow functions (e.g. running Ion on PyPy or using very large sample data) because it puts the larger number in the denominator, and we might also find that a proper statistical comparison is necessary to allow us to compare two sample means while taking into account the error from both samples.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue #, if available:
None
Description of changes:
PR-content-checkworkflowThis builds on #282 and must be merged after it. The perf workflow will not run successfully until this PR is rebased on top of #282. If you prefer, I can combine this into a single PR, but I thought it might be easier to review if I break them up.
You can see an example of the new workflow here. There is still some work to be done on getting the right metric for the performance benchmark comparison. I'm beginning to suspect that ops/s might be more stable for slow functions (e.g. running Ion on PyPy or using very large sample data) because it puts the larger number in the denominator, and we might also find that a proper statistical comparison is necessary to allow us to compare two sample means while taking into account the error from both samples.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.