Handshake performance of "reference code" vs "performance code"

[Honzaik]

Hi, while looking at the TLS handshake performance data at https://openquantumsafe.org/benchmarking/visualization/handshakes.html (specifically, from 2023-06-24 but other days have the same potential discrepancy) I came across a strange result. The reference code on x86 (and sometimes aarch64) is (almost) always faster (more handshakes per second) than the performance version for classical ECDHE (x25519/x448) with signature algorithm: Ed25519 or Dilithium2. With Ed25519:

• X25519 (x86_64-ref) | 1575.83
• X448 (x86_64-ref) | 1033.33

vs

• X25519 (x86_64-noport) | 1514.52
• X448 (x86_64-noport) | 1054.72 (here the performance code is faster)

or (aarch64)

• X25519 (aarch64-ref) | 1261.40
• X448 (aarch64-ref) | 608.65

vs

• X25519 (aarch64-noport) | 1252.21
• X448 (aarch64-noport) | 602.88

With Dilitihum2:

• X25519 (x86_64-ref) | 1320.63
• X448 (x86_64-ref) | 836.89

vs

• X25519 (x86_64-noport) | 1283.85
• X448 (x86_64-noport) | 814.40

When looking at post-quantum key exchanges it makes sense (performance code is faster than reference) but with classical key exchange it seems to be the opposite on x86.

However, when I select RSA2048 or ECDSAprime256v1 those results for ECDHE make sense (performance code is faster than reference) but then Kyber key exchange has the opposite effect (reference code Kyber + (RSA or ECDSA) is faster than performance code).

Is this a measurement error due to high variance? Is the "performance code" for classical key exchanges even different from the "reference code" since I would guess you use the OpenSSL implementation? Is there any other explanation for these strange results?

By the way, thank you for making and maintaining this project!

[baentsch]

Thanks for your interest(ed comments):

Is this a measurement error due to high variance?

This is my assumption, too. But I have no proof.

Is the "performance code" for classical key exchanges even different from the "reference code" since I would guess you use the OpenSSL implementation?

This is a very correct observation: Indeed, for plain, classic algorithms, there should be absolutely no performance difference as the figures for handshaking those algs comes straight from openssl s_time -- and that (code) indeed has been built outside of the "performance options" available in liboqs. My assumption indeed is that the VMs used for those tests have pretty high variance in load making these handshake results shaky (pardon the pun). What might be more tell-tale is running the tests on an empty, dedicated machine... If you have one, feel free to do that and check the numbers then.

Is there any other explanation for these strange results?

I found none. But admittedly, I have not looked long and hard at those numbers. You might have judged in sofar as well by seeing #74 and #97 still being open....

By the way, thank you for making and maintaining this project!

You're welcome. Glad to hear!