shotover / shotover-proxy

L7 data-layer proxy
https://docs.shotover.io
Apache License 2.0
84 stars 16 forks source link

chain microbenchmarks: run the chains once before benching to avoid measuring init time #1668

Closed rukai closed 3 months ago

rukai commented 3 months ago

While assisting with https://github.com/shotover/shotover-proxy/issues/1649 I noticed that our chain benches are slightly inaccurate in that they run the against a fresh transform chain every time. In a more realistic scenario the chain is reused across requests and the transforms can cache values internally after processing requests that can be reused on the following request.

Due to the way criterion works we cant reuse the chain from the previous iteration of the bench. So the best way to solve this is to send an unmeasured request batch through the chain during setup. This ensures that each iteration of the bench will operate on a chain which has had exactly one request batch pass through it already.

For the query_counter bench it will be useful to measure in both ways, so I've split it into query_counter_fresh and query_counter_pre_used. Which use a fresh chain and a pre used chain respectively.

For all other existing benches I converted them to use the pre used chain since that is most realistic.

codspeed-hq[bot] commented 3 months ago

CodSpeed Performance Report

Merging #1668 will not alter performance

Comparing rukai:chain_benches_prerun (2f5f934) with main (911e8a0)

Summary

✅ 37 untouched benchmarks

🆕 2 new benchmarks ⁉️ 1 dropped benchmarks

:warning: _Please fix the performance issues or acknowledge them on CodSpeed._

Benchmarks breakdown

Benchmark main rukai:chain_benches_prerun Change
⁉️ query_counter 13.8 µs N/A N/A
🆕 query_counter_fresh N/A 14.1 µs N/A
🆕 query_counter_pre_used N/A 14 µs N/A