ci: Continuous Benchmarking

[MarquessV]

Description

closes #1778

[MarquessV]

The PR benchmark workflow fails because there are no records for the default branch yet. That shouldn't be an issue after this PR lands and the first set of benchmarks get published.

[github-actions[bot]]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
7214	6326	88%	87%	🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: 61d2599 by action🐍

[Shadow53]

Thanks for those fixes!

I'm approving, but want to call out a couple of things:

1. The CI job is failing because there is no data for master. That is understandable, but I am concerned the job will break differently because that failure is hiding something else. Is it possible to commit some fake master data, e.g. from this branch, and use that for testing? Merging this should overwrite that data anyway, right?
2. Looking ahead, is there something we can use like CacheGrind to count instructions to avoid the issue of resource contention during benchmarking?

[MarquessV]

Thanks for those fixes!

I'm approving, but want to call out a couple of things:

1. The CI job is failing because there is no data for `master`. That is understandable, but I am concerned the job will break differently because that failure is hiding something else. Is it possible to commit some fake `master` data, e.g. from this branch, and use that for testing? Merging this should overwrite that data anyway, right?

2. Looking ahead, is there something we can use like `CacheGrind` to count instructions to avoid the issue of resource contention during benchmarking?

1. Maybe? bencher has a mock functionality that I might be able to use. I'll give it a try and if it's easy enough I'll go for it.
2. Not easily. None of the bencher supported adapters for Python support gathering that metric. Timing has its flaws, but my hope is that it will be consistent enough for the benchmarks we need in this project. At the very least, I think it makes sense to try it before investing more time in a custom solution that supports something like that.

[github-actions[bot]]

Bencher

Report	Fri, June 14, 2024 at 18:57:46 UTC
Project	pyquil
Branch	1778-implement-benchmarks-new
Testbed	ci-runner-linux

Click to view all benchmark results

Benchmark	Latency	Latency Results nanoseconds (ns) | (Δ%)	Latency Upper Boundary nanoseconds (ns) | (%)
test/benchmarks/test_program.py::test_copy_everything_except_instructions	✅ (view plot)	10,172,871,523.00 (-1.69%)	10,840,870,546.41 (93.84%)
test/benchmarks/test_program.py::test_instructions	✅ (view plot)	3,785,769,392.60 (-3.90%)	4,268,328,890.29 (88.69%)
test/benchmarks/test_program.py::test_iteration	✅ (view plot)	3,812,074,204.60 (-2.79%)	4,150,495,761.34 (91.85%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help