Performance measurements infrastructure

[arapov]

• What do we have today?
• Hardware needs?
• Where should we collect the data?
• What software will we use?
• Any framework for performance tests?
• Keep client and server stats in mind.
• We should be able to merge a PR that will be automatically taken for perf.test by infrastructure

[hlandau]

Possible ability to schedule coupled server and client on different machines to isolate measurements of server or client only

[hlandau]

Hardware

https://github.com/openssl/project/issues/137

We will buy physical servers.

Software

Thoughts:

• Some directory with performance test units, perhaps test/perf/...
  • TBD if we use the main repository or some other repository such as tools
• Infrastructure picks up on new performance test units as soon as it is merged to the repository we decide on, adds new graphs automatically
• Graph UI, hopefully publicly visible (Grafana maybe)
• Some data storage solution
• Some orchestration solution tying this all together
• Desirable to be able to run client and server on separate machines eventually but I don't think this is an "MVP" requirement

[t8m]

I am not sure why we cannot keep the performance tests in openssl/tools. IMO the tests should be largely version-independent. Possibly some of these will need some ifdefs or limitation of which versions they run against.

[mattcaswell]

I am not sure why we cannot keep the performance tests in openssl/tools

We want to be able to compare performance across different versions - so the tests need to be independent of any particular branch. Of course there may be some branches that a performance test is not relevant too (e.g. there is no point in measuring quic performance <3.2)

[hlandau]

OK, makes sense

[mattcaswell]

Something else to keep in mind: All outputs (i.e. graphs) need to be publicly accessible

[quarckster]

cc @nhorman

[nhorman]

my thoughts regarding what we need:

• A large machine (specs TBD, but I would think a 16/32 core system with 64-128GB ram minimum) to act as the test system
  • test system should be deployed as a self hosted github runner to tie it into CI, running a performance job regularly
  • a database (mysql, postgres, etc), deployed in GCP to hold persistent performance metrics
  • ci job running on test system should record perf metrics to mysql database
  • public grafana frontend to read database above and report test metrics
  • optional: tests in ci should be run multiple times, once in a 'raw' manner to record metrics without the impact of performance measurement tools, and multiple subsequent times to record whatever perf metrics we have with tools available to be recorded as ancillary data. I know this was discussed previously, but I feel strongly that gathering this ancillary data is an efficiency. It provides us with a way to correlate performance data without having to take the extra step of having to go through the process of local recreation.

[nhorman]

@quarckster can you provide a status update on this please? I know you were in the process of ordering this equipment, do we have it installed in the data center now? Can we start provisioning it?

[quarckster]

We are missing some hardware in our setup. We need to buy a network switch and a firewall. When everything will be installed I'll start to install the software.

[quarckster]

The firewall has arrived at the data center. The switch is on the way.

[quarckster]

New hardware set up epic https://github.com/openssl/private/issues/489

[quarckster]

I started looking into this task. What are we going to collect as performance data? Is it the output of perf applications from https://github.com/openssl/tools/ repo?

[quarckster]

I created an epic for this task https://github.com/openssl/project/issues/790