search

open-quantum-safe / liboqs

C library for prototyping and experimenting with quantum-resistant cryptography
https://openquantumsafe.org/
Other
1.88k stars 460 forks source link

Documenting build flags in test/speed config output #1145

Closed baentsch closed 2 years ago

baentsch commented 2 years ago

When looking at a test config output, not all build options can be seen and some have to be (but can be) inferred. This issue to ask whether all OQS_ build options should be explicitly output by https://github.com/open-quantum-safe/liboqs/blob/main/tests/system_info.c:

./tests/speed_kem Kyber512
Configuration info
==================
Target platform:  aarch64-Linux-5.11.0-1022-aws
Compiler:         gcc (10.3.0)
Compile options:  [-mcpu=native;-Werror;-Wall;-Wextra;-Wpedantic;-Wstrict-prototypes;-Wshadow;-Wformat=2;-Wfloat-equal;-Wwrite-strings;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]
OQS version:      0.7.0
Git commit:       b5cbb0d71a6f87f70198c986d85114beec1cb5ab
OpenSSL enabled:  Yes (OpenSSL 1.1.1f  31 Mar 2020)
AES:              OpenSSL
SHA-2:            OpenSSL
SHA-3:            C
CPU exts compile-time:  AES SHA2 NEON

doesn't show whether OQS_SPEED_USE_ARM_PMU had been set.

As a practical riddle along these lines: Who can explain/tell the performance difference in (terms of) configs between these two runs of speed_kem?

$ 
Configuration info
==================
Target platform:  aarch64-Linux-5.11.0-1022-aws
Compiler:         gcc (10.2.1)
Compile options:  [-mcpu=cortex-a72;-Werror;-Wall;-Wextra;-Wpedantic;-Wstrict-prototypes;-Wshadow;-Wformat=2;-Wfloat-equal;-Wwrite-strings;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]
OQS version:      0.7.0
Git commit:       b5cbb0d71a6f87f70198c986d85114beec1cb5ab
OpenSSL enabled:  Yes (OpenSSL 1.1.1k  25 Mar 2021)
AES:              OpenSSL
SHA-2:            OpenSSL
SHA-3:            C
CPU exts compile-time:  NEON

Speed test
==========
Started at 2021-12-06 13:47:34
Operation                      | Iterations | Total time (s) | Time (us): mean | pop. stdev | CPU cycles: mean          | pop. stdev
------------------------------ | ----------:| --------------:| ---------------:| ----------:| -------------------------:| ----------:
Kyber512                       |            |                |                 |            |                           |           
keygen                         |     131591 |          3.000 |          22.798 |     22.217 |                     13164 |      21927
encaps                         |     113083 |          3.000 |          26.529 |      1.250 |                     37924 |      29535
decaps                         |     129931 |          3.000 |          23.089 |      0.815 |                     31669 |      26208
Ended at 2021-12-06 13:47:43

$ 
Configuration info
==================
Target platform:  aarch64-Linux-5.11.0-1022-aws
Compiler:         gcc (10.2.1)
Compile options:  [-Werror;-Wall;-Wextra;-Wpedantic;-Wstrict-prototypes;-Wshadow;-Wformat=2;-Wfloat-equal;-Wwrite-strings;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]
OQS version:      0.7.0
Git commit:       b5cbb0d71a6f87f70198c986d85114beec1cb5ab
OpenSSL enabled:  Yes (OpenSSL 1.1.1k  25 Mar 2021)
AES:              OpenSSL
SHA-2:            OpenSSL
SHA-3:            C
CPU exts active:  AES SHA2 NEON
Speed test
==========
Started at 2021-12-06 13:48:19
Operation                      | Iterations | Total time (s) | Time (us): mean | pop. stdev | CPU cycles: mean          | pop. stdev
------------------------------ | ----------:| --------------:| ---------------:| ----------:| -------------------------:| ----------:
Kyber512                       |            |                |                 |            |                           |           
keygen                         |      63667 |          3.000 |          47.121 |     18.503 |                     63673 |      52375
encaps                         |      51177 |          3.000 |          58.621 |      3.699 |                     83904 |      65434
decaps                         |      43890 |          3.000 |          68.353 |      3.427 |                     97736 |      78182
Ended at 2021-12-06 13:48:28

???

dstebila commented 2 years ago

When looking at a test config output, not all build options can be seen and some have to be (but can be) inferred. This issue to ask whether all OQS_ build options should be explicitly output by https://github.com/open-quantum-safe/liboqs/blob/main/tests/system_info.c:

It seems like that would be worthwhile to avoid ambiguity / confusion.

As a practical riddle along these lines: Who can explain/tell the performance difference in (terms of) configs between these two runs of speed_kem?

$ 
Configuration info
==================
Target platform:  aarch64-Linux-5.11.0-1022-aws
Compiler:         gcc (10.2.1)
Compile options:  [-mcpu=cortex-a72;-Werror;-Wall;-Wextra;-Wpedantic;-Wstrict-prototypes;-Wshadow;-Wformat=2;-Wfloat-equal;-Wwrite-strings;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]
OQS version:      0.7.0
Git commit:       b5cbb0d71a6f87f70198c986d85114beec1cb5ab
OpenSSL enabled:  Yes (OpenSSL 1.1.1k  25 Mar 2021)
AES:              OpenSSL
SHA-2:            OpenSSL
SHA-3:            C
CPU exts compile-time:  NEON

Speed test
==========
Started at 2021-12-06 13:47:34
Operation                      | Iterations | Total time (s) | Time (us): mean | pop. stdev | CPU cycles: mean          | pop. stdev
------------------------------ | ----------:| --------------:| ---------------:| ----------:| -------------------------:| ----------:
Kyber512                       |            |                |                 |            |                           |           
keygen                         |     131591 |          3.000 |          22.798 |     22.217 |                     13164 |      21927
encaps                         |     113083 |          3.000 |          26.529 |      1.250 |                     37924 |      29535
decaps                         |     129931 |          3.000 |          23.089 |      0.815 |                     31669 |      26208
Ended at 2021-12-06 13:47:43

$ 
Configuration info
==================
Target platform:  aarch64-Linux-5.11.0-1022-aws
Compiler:         gcc (10.2.1)
Compile options:  [-Werror;-Wall;-Wextra;-Wpedantic;-Wstrict-prototypes;-Wshadow;-Wformat=2;-Wfloat-equal;-Wwrite-strings;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]
OQS version:      0.7.0
Git commit:       b5cbb0d71a6f87f70198c986d85114beec1cb5ab
OpenSSL enabled:  Yes (OpenSSL 1.1.1k  25 Mar 2021)
AES:              OpenSSL
SHA-2:            OpenSSL
SHA-3:            C
CPU exts active:  AES SHA2 NEON
Speed test
==========
Started at 2021-12-06 13:48:19
Operation                      | Iterations | Total time (s) | Time (us): mean | pop. stdev | CPU cycles: mean          | pop. stdev
------------------------------ | ----------:| --------------:| ---------------:| ----------:| -------------------------:| ----------:
Kyber512                       |            |                |                 |            |                           |           
keygen                         |      63667 |          3.000 |          47.121 |     18.503 |                     63673 |      52375
encaps                         |      51177 |          3.000 |          58.621 |      3.699 |                     83904 |      65434
decaps                         |      43890 |          3.000 |          68.353 |      3.427 |                     97736 |      78182
Ended at 2021-12-06 13:48:28

???

I see that AES and SHA2 extensions are active on the latter, which *checks notes* makes things slower?

baentsch commented 2 years ago

It seems like that would be worthwhile to avoid ambiguity / confusion.

Working on it.

I see that AES and SHA2 extensions are active on the latter, which checks notes makes things slower?

Weird, AES and SHA2 extensions present that make things slower?!? Shouldn't stuff with extensions available be faster? We might at least want to phrase the output differently....

baentsch commented 2 years ago

@dstebila Could you please take a look at branch mb-configupdate? This lists/explains all build options in the comment and adds the missing output as and when branch 'ds-pmu-option' lands. A better suggestion as to how to state presence/absence of OQS_DIST_BUILD still solicited.

dstebila commented 2 years ago

@dstebila Could you please take a look at branch mb-configupdate? This lists/explains all build options in the comment and adds the missing output as and when branch 'ds-pmu-option' lands. A better suggestion as to how to state presence/absence of OQS_DIST_BUILD still solicited.

Regarding the various OQS_DIST_*BUILD flags, we could just print that out directly too — although parts of it can be inferred from other output, that assumes we've got all the rules correct everywhere, so it might be helpful for debugging purposes to directly output which of those is set. Same for OQS_DEBUG_BUILD. Looking through oqsconfig.h, there's also OQS_ENABLE_TEST_CONSTANT_TIME.

baentsch commented 2 years ago

so it might be helpful for debugging purposes to directly output which of those is set. Same for OQS_DEBUG_BUILD. Looking through oqsconfig.h, there's also OQS_ENABLE_TEST_CONSTANT_TIME.

Well, oqsconfig.h is an awfully long list (258 entries if grep didn't lie :), so outputting everything wouldn't be sensible. My proposal was to ensure all options documented in the wiki shall be made visible (by direct or indirect outputs). I agree, it may be reasonable to also document OQS_ENABLE_TEST_CONSTANT_TIME there. If you'd want to take a pass at completing the Wiki, I'll follow up amending this branch and do a PR.

dstebila commented 2 years ago

oqsconfig.h is long, but most of that is algorithm enable lines, which aren't relevant to this discussion. The only other one that seemed relevant was OQS_ENABLE_TEST_CONSTANT_TIME. I've now added that to the Wiki.