Continuous performance regression testing

[achamayou]

It would be good for the project to have a few end to end benchmarks at least, and to run them regularly to at least spot performance regressions quickly, if not prevent them.

A setup similar to what CCF does may be a good place to start: bencher infra, CI job, Dashboard.

[andpiccione]

@achamayou We already have a few performance tests that run during the CI:

• https://github.com/microsoft/scitt-ccf-ledger/blob/main/test/test_perf.py
• https://github.com/microsoft/scitt-ccf-ledger/blob/main/test/test_load.py

Of course it would be great to have a more thorough end-to-end benchmark suite like CCF does, but the current tests could be good enough to spot most of the performance regressions.

[achamayou]

@andpiccione I found those but I am not sure how realistic they are compared to the scenarios we are discussing. I can try to create a bencher project and hook them up if you would like.