Track performance of ztunnel in CI

[howardjohn]

We have benchmarks but no tracking.

Requirements:

• Presubmit should have a way to run against HEAD and compare results.
  • If this is slow/expensive, we can make it on-demand
• Postsubmit should compare HEAD to HEAD^. Regressions should fail
• We should have a graph of each commit to track patterns over time
• Testing should not have noise
  • This might require running both commit versions we are comparing in one job, rather than comparing against old results, to have apples to apples machines.

[PiotrSikora]

Doesn't this pretty much require dedicated metal for CI?

[howardjohn]

If we run HEAD and HEAD^ on the same machine it might be sufficient, assuming there is minimal noise within a single machine. We can get a dedicated node in prow so there is at least only our single job running on it. Although that makes the graph not really work.

[PiotrSikora]

I'm getting +/- 5% between subsequent runs on the same VM, and that granularity would hide small regressions.

[howardjohn]

Bummer... we could try increasing sample sizes/iterations in criterion but if there is enough noise it may still be tricky.

I think its important to have some way to make sure performance is not regressing. That may not be easy though, and may end up looking different than "run the job in prow" I suppose

[jaellio]

@Stevenjin8

[Stevenjin8]

Bullets 1 and 4 (ish) are done through #559 and #551, but tests are very noisy.