Parameterized docker image

[baentsch]

Given this repo contains most work to bring OQS and QUIC-TLS together, would it be the right place to add a Dockerfile to allow for more easy parameterization and experimentation along the lines documented at the end of the published bash script? If you'd agree @igorbarshteyn, I'd give this a first try at implementing, maybe afterwards moving things over to OQS-demos.

[igorbarshteyn]

Hi Michael,

Sure! I would very much welcome it.

A couple of notes regarding how this demo was built, in case parts need to be reverse engineered or updated:

1) I started with the December release candidate of oqs-openssl (which synced up with OpenSSL 1.1.1m). I then went over to the quictls github and used the "git cherry" command to bring over the changes at the tip of their OpenSSL 1.1.1m+quic branch to oqs-openssl. quictls made this easy since they put all the QUIC changes at the very tip of the branch and marked them with "QUIC: ..." - this process can be easily repeated for other versions as long as the underlying OpenSSL version numbers match between oqs-openssl and quictls.

2) Regarding the lsquic build that references oqs-boringssl: In the demo I provide it as a pre-built download, because automating the build in my script was a pain. To get lsquic to build, I had to note each error reported by cmake while following the build instructions for lsquic (but substituting oqs-boringssl for regular BoringSSL). Each error was corrected by changing build.make and link.txt files in each failing object's CMakeFiles directory, to append references to liboqs.a (from liboqs that was used to build oqs-boringssl referenced in the lsquic build, so in my case the path was /home/user/liboqs_B/build/lib/liboqs.a). These liboqs.a entries were positioned just after the references to libcrypto.a, to get the build to complete.

3) The nginx-quic build uses a customized Makefile which the demo script downloads to overwrite the Makefile created with the "configure" command for nginx-quic. The reasons for this are a) that the paths for include subdirectories pointing to oqs-openssl-quic in the original Makefile don't work - there is an extra .openssl entry in the path that is unnecessary b) the called oqs-openssl-quic build needed to be configured to include all the available KEX groups, and c) a reference to liboqs.a needed to be added to the Makefile to get nginx-quic to build*.

*Note: While writing you this message I was going over my script and found that line 78 had a bug in the configure statement for the nginx-quic build: the path for the --with-openssl argument referenced "oqs-openssl/quic" instead of "oqs-openssl-quic". The error was masked by the customized Makefile and everything still builds anyway, but just to make sure we have clean code I fixed this up and retested just now, and uploaded the fixed version to AWS. This was the only change.

I hope this helps, and please let me know if you have any questions.

Thanks,

Igor

[igorbarshteyn]

Michael,

P.S.> I invited you to this repository.

Thanks,

Igor

[baentsch]

Hi Igor,

thanks for the invite (gladly accepted) and the additional information above.

To make this proposal even more meaningful I thought about the following:

• Create one "server-side" (ubuntu) build script only meant to set up a QSC-TLS-QUIC nginx server opening one port each per QSC algorithm combination (much like https://test.openquantumsafe.org) and meant to be run on some cloud provider VM (that permits QUIC data to pass through).
• Create one "client-side" docker image allowing anyone to run a QSC-TLS-QUIC client, particularly tuned towards running performance tests towards the above test server

--> This way we could collect "real-world" measurements for all algorithms from different vantage points in the world without the need to make people build the full software stack or analyze wireshark traces. What'd you say, interesting/meaningful?

One question towards your lsquic build: Was this necessary because quic-openssl does not support client-side operation for QUIC (i.e., s_client/s_time) command or because you wanted to exchange more data than just a handshake? This question basically is meant to ask the question: Does oqs-quic-openssl permit executing s_time ? If so, would you think one could leave away lsquic, oqs-boringssl (with its functionality limitations) and wireshark from the client-side image and simply "measure performance"? In a further extension to this idea, I'd envision a patched version of oqs-quic-openssl to count (and output) retransmissions in order to show how well (or bad) different QSC algorithms are performing in this regard. I kind of expect performance numbers to closely correlate with such retransmission figures (i.e., explaining them with concrete figures for each algorithm combination). Would you agree?

[igorbarshteyn]

Hi Michael,

I think your idea of splitting up the client and server is excellent for the purposes of real-world performance testing. It would allow for testing over actual wire, rather than just locally, and would be a better simulation of what would happen to fragmented packets and the delays in reassembly and handshakes.

Regarding the lsquic build: I went with this because part of my original motivation for the project was to build a working system that displayed the received contents of the webpage being served. lsquic http/3 client allowed for that possibility. Additionally, I wasn't sure if I could tweak the QUIC MTU parameters with s_client, whereas it was possible with lsquic with the -z command line parameter. The loss attendant to going down this path, of course, was that I couldn't test hybrid signatures since oqs-boringssl didn't support them. Lastly, while adding the QUIC-related commits to oqs-openssl enabled QUIC protocol support in the library, I didn't do anything to modify s_client itself, so wasn't sure that it could "speak QUIC" despite the support for the QUIC API's being there. I figured if I take a dedicated HTTP3/QUIC client and build it against oqs-openssl-quic, then everything would fall into place and (based on my limited testing), at least p256_bikel1 over RSA-2048/falcon1024 did seem to work...

During my build out testing I did test s_client against s_server in oqs-openssl-quic, and it seemed to work. I didn't do anything quantum-safe, just regular testing against RSA-2048 with X25519, to confirm that those pieces built OK. s_client was running and not crashing out, so I noted that as a positive result and moved on to building lsquic.

I can try to spin up my test box and try s_client this evening against the nginx-quic server and let you know if it actually connects, but I have my doubts since I don't think the code is there for it to negotiate QUIC connections itself...

Thanks,

Igor

[baentsch]

I can try to spin up my test box and try s_client this evening against the nginx-quic server and let you know if it actually connects, but I have my doubts since I don't think the code is there for it to negotiate QUIC connections itself...

Much appreciated! Could you then please also check out s_time before we go down that path? Will begin until then following your script to get a "server-setup-only"...

[igorbarshteyn]

Hi Michael,

I ran the tests. As I surmised, oqs-openssl-quic s_client speaks to its s_server just fine, with what looks like any combination of post-quantum or conventional KEX or signature offered in openssl-oqs. I enclose a sample screenshot of test output between the two using firesaber and falcon1024 for KEX and signature, respectively.

However, out of the box it looks like oqs-openssl-quic doesn't know how to build QUIC sockets (?). When I run s_client against the nginx-quic server configured in a similar fashion, it fails (output enclosed) with an error message saying connection refused. A similar outcome happens with s_time (output also enclosed).

To sanity-check myself, I also ran a spot test using lsquic against nginx-quic using firesaber and falcon1024 and it failed (with a failed handshake). However, rerunning this with p256_bikel1 and falcon1024 works (ZIPped packet capture enclosed). It looks like "longer" KEXs seem to break things, but I will have to wait to determine which ones work until I do more spot testing.

So bottom line: it looks like s_client and s_time won't work, unless they are rewritten to "speak QUIC", whereas lsquic build with oqs-boringssl works just fine.

lsquic vs nginx-quic with p256_bikel1 and falcon1024 cert.zip

[baentsch]

Hi Igor,

thanks for these tests. Some comments/questions:

As I surmised, oqs-openssl-quic s_client speaks to its s_server just fine, with what looks like any combination of post-quantum or conventional KEX or signature offered in openssl-oqs.

Very well -- the key question is: Is it doing this using TCP or QUIC? If the former, the subsequent problems with quic-nginx would be explained (but then I don't understand how/why quictls could truly be considered a "QUIC-enabled OpenSSL"). If running QUIC, the issue might be something else we'd need to look into.

To sanity-check myself, I also ran a spot test using lsquic against nginx-quic using firesaber and falcon1024 and it failed (with a failed handshake).

That I would expect as I understand lsquic to be based on oqs-boringssl (?) where the client by default only requests p256_ hybrid KEMs, so only "p256_lightsaber" would work "out of the box".

[baentsch]

Ouch -- I only now (think I) truly understood what quictls does: From its README:

This fork adds APIs that can be used by QUIC implementations for connection handshakes.

In other words, it doesn't do QUIC "inside". This is also supported by the statement

[What about ] ...and the executable? We currently do not have any plans to change the name, mainly because we haven't made any changes there.

This reads to me that the executables' functionality is unchanged (i.e., using TCP). Accordingly, one would need to use MSQuic or Chromium QUIC.

So my new question: Why did you endeavour to use lsquic instead (of say msquic) for client-side operation?

[LPardue]

FWIW in case it helps here, https://interop.seemann.io/ is a service we used for continuos automated testing between QUIC clients and servers running in Docker containers. Lots of info in the readme at https://github.com/marten-seemann/quic-interop-runner/blob/master/README.md#building-a-quic-endpoint

[dstebila]

FWIW in case it helps here, https://interop.seemann.io/ is a service we used for continuos automated testing between QUIC clients and servers running in Docker containers. Lots of info in the readme at https://github.com/marten-seemann/quic-interop-runner/blob/master/README.md#building-a-quic-endpoint

Tagging @xvzcf as he's been developing a TLS interop runner so he might be interested.

[igorbarshteyn]

Hi Everyone,

I've posted results of initial unit testing for key exchanges and signatures on LinkedIn (https://www.linkedin.com/feed/update/urn:li:ugcPost:6888334865943547904?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A6888334865943547904%2C6889577876790767617%29) and also reproduce them below for your convenience.

I'll respond to Michael's questions and comments tonight after I sign off at work.

---

Hi Everyone,

I finally have some first-pass data to share!

I'm happy to let you know that I've completed unit testing of individual key-exchanges and signatures using this platform, and found some interesting results.

After confirming that the "baseline" configuration with an X25519 key exchange and an RSA-2048 certificate works in completing the TLS 1.3 handshake for QUIC, I tested quantum-safe and hybrid key exchanges paired with an RSA-2048 server certificate, between the lsquic client and nginx-quic server.

Next, I tried to use oqs-openssl-quic to generate certificates with quantum-safe signatures, and then to use these certificates on the nginx-quic server and try for a handshake with the lsquic client using an X25519 key exchange. For each test, I noted whether the handshake completed between client and server, and for certificates, whether they were able to be generated, and if so, whether the handshake completed. Here is a quick summary of my results:

Key Exchanges - it seems like only a subset of hybrid key exchanges works, and they are all hybridized with the NIST-P256 curve. All other handshakes fail (it could be that their curves are not being properly advertised by either the client or server, because the handshakes failed in just two packets - not due to fragmentation or delays - I am investigating that part). It is interesting to note that not all P256 hybrids work, but only some. Working key exchanges are:

• p256_bikel1 • p256_kyber90s512 • p256_frodo640aes • p256_ntru_hps2048509 • p256_lightsaber • p256_sidhp434 • p256_sikep434

Signature Algorithms - Dilithium and Dilithium/AES, Falcon, and Rainbow I Classic generate certificates, and work to complete TLS handshakes over QUIC. Rainbow V classic causes the fan on my laptop to go into high RPM mode but does generate a certificate, though the handshake with this certificate never completes (just goes into an infinite loop of handshakes with no resulting session establishment). As for the others, only a couple of picnic variants and three sphincs combination signatures work. The majority of signature algorithms fail to generate a certificate with an "unknown algorithm" error. Here is the list of signature algorithms that successfully generated certificates and completed the handshake:

• dilithium2 • dilithium3 • dilithium5 • dilithium2_aes • dilithium3_aes • dilithium5_aes • falcon512 • falcon1024 • picnicl1full • picnic3l1 • rainbowIclassic • sphincsharaka128frobust • sphincssha256128frobust • sphincsshake256128frobust

In summary, I think there are a couple of issues that may be impairing the functionality of my build:

a) The key exchange algorithms may not all be getting advertised by either the client or server. b) Certificate generation is broken somewhere in the build, and only "likes" certain algorithms.

I think the above two issues may be due to problems with the software vs QUIC itself. However, completed handshakes in testing did show performance discrepancies between different key exchange and signature algorithms, due to varying degrees of packet fragmentation due to QUIC MTU limitations. You can review the results by checking out the enclosed packet capture ZIP files below.

I enclose a summary table of my results, and packet capture ZIP files for key exchange and signature tests (where the certificates generated), below.

• https://li-article-bucket.s3.us-east-2.amazonaws.com/oqs-openss-quic_testing.xlsx • https://li-article-bucket.s3.us-east-2.amazonaws.com/oqs-openssl-quic_KEX_testing.zip • https://li-article-bucket.s3.us-east-2.amazonaws.com/oqs-openssl-quic_SIG_testing.zip

Thanks everyone!

--Igor

[baentsch]

Thanks again for all these results. Quick feedback:

Key Exchanges - it seems like only a subset of hybrid key exchanges works, and they are all hybridized with the NIST-P256 curve. All other handshakes fail

That at most those work is expected as per https://github.com/open-quantum-safe/boringssl/wiki/Implementation-Notes (item 2).

[igorbarshteyn]

Hi Michael,

In reply to your previous question about s_client: you are correct - quictls doesn't do QUIC "inside" - it just supports QUIC calls from other clients. This is exactly the feature I wanted to add to oqs-openssl in my fork, so I could build a QUIC client against it, which could make QUIC calls that it would "understand" in addition to quantum-safe algorithms.

To confirm further I ran another test of oqs-openssl-quic s_client against nginx-quic, and captured the packets. s_client sends things over TCP, whereas the nginx-quic server in my configuration has TCP specifically disabled (in the default configuration, the nginx-quic team recommends to use the same port to accept QUIC and TCP requests for wider compatibility, but I wanted to ensure requests are accepted over QUIC only for testing and to avoid confounding results caused by "backwards compatibility"). s_client to nginx-quic (p256_bikel1 with RSA-2048).zip

In response to your question about lsquic: I considered a number of clients, and my primary considerations were ability to be built against openssl, and ability to customize connection parameters to the greatest extent possible at the command line, while connecting to an http/3 server like nginx-quic. I tried ngtcp2 and it wouldn't build properly because of the ngttp3 middleware not building with oqs-openssl-quic. I did look at msquic, and I saw it could be configured to build against openssl as its TLS provider, but I couldn't figure out if there was a client packaged in msquic that I could use to test against the nginx-quic server. It looked like there were a number of PowerShell scripts for testing different QUIC functionalities, but no complete client that could be used to test webserver connectivity. Didn't look at Chromium QUIC - was worried it would be a pain to build like Chromium itself is - I have an older laptop I'm doing this on, and it doesn't have the specs needed to do a real Chromium build in less than a week - that would make iteration impossible. I'll take a look though - maybe it's smaller than the browser...

Regarding the boringssl KEX limitations: understood - that makes me feel better regarding my build. I'd love to know where to change the kDefaultGroups value so I can rebuild boringssl to expand the list of groups announced to match that in oqs-openssl-quic for this experiment.

I'm still perplexed about certificate generation failing with "unknown algorithm" for most signature algorithms; I am using the openssl command line to generate those certificates so it should work, but it doesn't. That may indeed be a problem with my build.

Thanks again!

Igor

[baentsch]

Hi Igor, thanks for the re-confirmation of my understanding pertaining to "no-QUIC-inside-quic-openssl".

I'd love to know where to change the kDefaultGroups value so I can rebuild boringssl to expand the list of groups announced to match that in oqs-openssl-quic for this experiment

This is documented in the Wiki link I sent above (click on the "extensions.cc" link).

I'm still perplexed about certificate generation failing with "unknown algorithm" for most signature algorithms; I am using the openssl command line to generate those certificates so it should work, but it doesn't. That may indeed be a problem with my build.

That would be surprising (assuming you ran the oqs-tests -- did you?), but then again... What about trying cert-generation with our pre-built openquantumsafe/curl docker image? Can you please specify the exact command you issue trying to create the certs? I'll then check those when putting together a "quic-enabled" oqs-docker image to ascertain that works and we can focus on true QUIC issues.

[igorbarshteyn]

Hi Michael,

I'm closing out some major projects against deadlines at work. My apologies, but I will only be able to reply over the weekend, at the level of detail to do justice to the points you raised.

Thank you for your patience,

Igor

[igorbarshteyn]

Hi Michael,

I ran the oqs-tests full test as well as the CMS test, and everything passed on my test machine using oqs-openssl-quic. I'm enclosing the piped outputs I got from each command in a ZIP file for your review.

As for the commands I ran to generate the certificates:

I used the accessory script https://li-article-bucket.s3.us-east-2.amazonaws.com/recert.sh to generate the certificates for my tests. It's entirely possible I used the wrong command syntax structure and/or that I used incorrect signature algorithm names. I modeled the code in the script on the code in https://github.com/open-quantum-safe/openssl#tls-demo, simply replacing "SIG" with the $SIGCIPHER variable, where in each test I set the value to the next signature algorithm name and ran the script to regenerate the certs. I took the list of signature algorithm names for my tests from https://github.com/open-quantum-safe/openssl#supported-algorithms, and these are the same names reflected in my results spreadsheet found at https://li-article-bucket.s3.us-east-2.amazonaws.com/oqs-openss-quic_testing.xlsx. Maybe the hybrid ones should have been ECDSA certificates with that second command as described in https://github.com/open-quantum-safe/openssl#tls-demo ... I'm not clear on this point?

I'll try to generate with the open-quantum-safe/curl docker image, and retest, but my exposure to docker is limited so I'll need a bit of time to learn how to deploy it on my test box.

Thanks!

--Igor oqs-test-output.zip

[baentsch]

I ran the oqs-tests full test as well as the CMS test, and everything passed on my test machine using oqs-openssl-quic. I'm enclosing the piped outputs I got from each command in a ZIP file for your review.

They all look good.

these are the same names reflected in my results spreadsheet found at https://li-article-bucket.s3.us-east-2.amazonaws.com/oqs-openss-quic_testing.xlsx.

Thanks for this spreadsheet! The first column (Key Exchange) is exactly aligned with https://github.com/open-quantum-safe/boringssl/blob/master/ssl/extensions.cc#L383-L394, so completely as expected (if one relies on the default algorithms announced and doesn't specify the ones to be used explicitly). The second column (signatures) is very surprising as your generation script looks fine to me and things are OK for many algorithms: Could you share logs with the error messages to review (which command fails how)?

I'm not clear on this point?

Please check out https://github.com/open-quantum-safe/openssl#authentication: What should always work for hybrids is prefixing "p256", "p384" or "p521_" to each QSC signature algorithm name, depending on its NIST strength as documented for each algorithm family e.g., here (check out the .md files for easier readability): NIST level 1&2 are considered "1"/p256, 3&4 equate "3"/p384 and NIST level 5 is "5"/p521. I'll probably update the oqs-openssl documentation in this regard.

I'll try to generate with the open-quantum-safe/curl docker image, and retest, but my exposure to docker is limited so I'll need a bit of time to learn how to deploy it on my test box.

We have created a development image for this to make things (hopefully) straightforward to work with: If you just run docker run -it openquantumsafe/curl-dev sh you'll find yourself as root in a "normal" alpine shell environment where everything (oqs-curl, oqs-openssl and liboqs) is build, tested to work OK and installed in /opt/oqssa.

[baentsch]

It'd be great if you could take a look at the README of https://hub.docker.com/repository/docker/openquantumsafe/nginxquic: It enables experimentation with all OQS algorithm combinations (each SIG/KEM combination with its own port).

[baentsch]

Closed via https://github.com/open-quantum-safe/oqs-demos/pull/123 and https://github.com/open-quantum-safe/oqs-demos/pull/124