oqs-quic-openssl client

[baentsch]

Allow me to split out the discussion from https://github.com/igorbarshteyn/oqs-openssl-quic/issues/1#issuecomment-1016956697:

my primary considerations were ability to be built against openssl, and ability to customize connection parameters to the greatest extent possible at the command line, while connecting to an http/3 server like nginx-quic

Those selection criteria are excellent -- I would personally compromise config parameters missing in exchange for a client fully working with openssl (APIs), though.

I tried ngtcp2 and it wouldn't build properly because of the ngttp3 middleware not building with oqs-openssl-quic. EDITED TO ADD: It looks like I recalled this incorrectly - pulling up my notes (per below) it was an issue with ngtcp2 not detecting QUIC support in oqs-openssl-quc (see below)

Did you takes notes what you tried and what failed? Maybe I could take a second look? Maybe worth while discussing in a dedicated issue?

I did look at msquic, and I saw it could be configured to build against openssl as its TLS provider, but I couldn't figure out if there was a client packaged in msquic that I could use to test against the nginx-quic server. It looked like there were a number of PowerShell scripts for testing different QUIC functionalities, but no complete client that could be used to test webserver connectivity.

Same request: Any links to share? ~If it's truly only building/useful on Windows that would be a bummer of course...~ According to msquic it is supported under Linux using exactly this code: https://github.com/microsoft/msquic/blob/main/docs/Platforms.md#linux

Further, this sample client code in my opinion would serve most purposes, no? No full HTTP/3 but enough to ascertain how/whether QSC connections can be established.

it would be a pain to build like Chromium itself is

That I can fully understand. For such cases I'm firing up a pretty big AWS VM that churns through this comparatively fast -- but again, maybe it isn't such a "beast", so indeed, if you could take a look, that might be helpful.

All the above would be to work around the limitations of oqs-boringssl in terms of QSC functionality, of course: Apologies we never got around to bring it to the same level as oqs-openssl, but we simply had more use(r)s for the latter....

In sum, my question: What about focusing on using msquic for establishing (or not :) QSC-QUIC connections with the oqs-quic-nginx server you built? Doable? Sensible?

[igorbarshteyn]

Hi Michael,

I'm closing out some major projects against deadlines at work. My apologies, but I will only be able to reply over the weekend, at the level of detail to do justice to the points you raised.

Thank you for your patience,

Igor

[baentsch]

No rush, please. Apologies if it seems like I'm (overly) "activist": Not my intention -- My curiosity just sometimes "runs wild" :).

[igorbarshteyn]

Hi Michael,

I took notes and saved all my test scripts so I was able to recreate the issue I faced with ngtcp2.

Below, you will find the block of script code I used for building ngtcp2 (which, back at that iteration of oqs-openssl-quic.sh, was used instead of steps 7 and 8 in the currently published script).

The issue I found was that the configure script for ngtcp2, while successfully detecting the presence of OpenSSL at the location for oqs-openssl-quic, consistently failed to detect QUIC support (see screenshot of the configure script output). The build documentation for ngtcp2 insisted that it was compatible with quictls (which is where I cherry-picked all the QUIC support commits from), so I was perplexed as to why QUIC wasn't detected.

I dug a little deeper into the configure script, and located the code block where the check for QUIC support happens (see 2nd screenshot), but since I'm not a developer but just an IT/security person, I wasn't able to figure out exactly what it was checking for. I don't exclude that my library or include paths were wrong - although I felt that the package configure parameters should have taken care of that? I even tried to "hack" the configure script to always return 'yes' for QUIC features being present anyway and tried to build it like that. I would then hit undefined reference errors to OQS components (which was something I saw in other client build attempts and was previously able to fix by adding references to liboqs.a where appropriate). In this case, the makefiles for ngtcp2 were too convoluted for me to decipher with my amateur eye and I couldn't figure out where to make the necessary changes to complete the build.

After this digression I determined it was too "dirty" of an attempt anyway, with too much brute hacking instead of proper configuration from the start, and I should just abandon ngtcp2 altogether. Perhaps someone like you could figure out what the ngtcp2 configure script was checking for in terms of QUIC, and why it wasn't finding it. Hopefully it's just a path problem.

Thanks!

--Igor

Step 'X': Download and install ngtcp2 QUIC client and build against custom OpenSSL version

sudo apt install pkg-config autoconf automake autotools-dev libtool libev-dev libcunit1-dev git clone https://github.com/ngtcp2/nghttp3 cd nghttp3 autoreconf -i ./configure --prefix=$HOME/oqs-openssl-quic --enable-lib-only make -j 1 check make install cd .. git clone https://github.com/ngtcp2/ngtcp2 cd ngtcp2 autoreconf -i ./configure PKG_CONFIG_PATH=$HOME/oqs-openssl-quic/build/lib/pkgconfig:$HOME/nghttp3/build/lib/pkgconfig make -j 1 check

[igorbarshteyn]

Hi Michael,

Also, with respect to msquic, I did see that it builds in Linux. I could probably give that a try tomorrow evening after the kids go to bed and see how much progress I make :) However feel free to preempt me if you so choose.

Thanks,

Igor

[baentsch]

Not pre-empting anything, but when looking into the problem, it seems the autoconfig test program of ngtcp2 checking for quic-openssl presence doesn't get properly compiled because the link options "-ldl -pthread" are missing in openssl's libcrypto.pc file (rather, are defined only private and not global), thus returning "no" (quic support). I need to check which code introduced this erroneous declaration...

[igorbarshteyn]

Hi Michael,

The way Microsoft packages MSQUIC is they package quictls as a git submodule, against which it builds with cmake.

I gave it a try to build MSQUIC "in a vacuum" with oqs-openssl-quic instead, on a fresh Ubuntu machine. Here is my bash script enclosed as a ZIP file, to try and accomplish it. In a nutshell, you have to start with MSQUIC pre-release 1.9 (it supports quictls). Then you clone it, but replace the openssl submodule with oqs-openssl-quic. Then you get liboqs, build it and install it into the submodule directory for oqs-openssl-quic. Then build MSQUIC.

I still hit a problem in the middle of the build with undefined references to oqs.h or liboqs.a - need to look into those and hunt those down. But this seems to be the process I'm trying to follow so far. Could be a better way to do it, but I haven't found it yet, after reading the MSQUIC docs.

msquic.zip

[baentsch]

Hi Igor, thanks for this -- this basically does what the msquic-building Dockerfile I published last week tries to accomplish: Unfortunately, this branch creates the very same core dump I already got when running build/bin/Release/msquicplatformtest -- also when leaving away OQS completely:

Here's the script to reproduce what I now did:

#! /bin/bash

cd /root

apt-add-repository ppa:lttng/stable-2.12 -y && apt update && apt install build-essential cmake liblttng-ust-dev lttng-tools git ninja-build libssl-dev -y

git clone --branch prerelease/1.9 https://github.com/microsoft/msquic && cd msquic && git submodule init && git submodule update --recursive

cd /root/msquic && mkdir build && cd build && cmake -G 'Unix Makefiles' ..  && cmake --build .

results in:

root@96a2d3cead8c:~# ./msquic/build/bin/Release/msquicplatformtest 
[==========] Running 70 tests from 7 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from CryptTest
[ RUN      ] CryptTest.WellKnownClientInitial
[       OK ] CryptTest.WellKnownClientInitial (0 ms)
[ RUN      ] CryptTest.WellKnownChaChaPoly
[       OK ] CryptTest.WellKnownChaChaPoly (0 ms)
[ RUN      ] CryptTest.HpMaskChaCha20
[       OK ] CryptTest.HpMaskChaCha20 (0 ms)
[ RUN      ] CryptTest.HpMaskAes256
[       OK ] CryptTest.HpMaskAes256 (0 ms)
[ RUN      ] CryptTest.HpMaskAes128
[       OK ] CryptTest.HpMaskAes128 (0 ms)
[----------] 5 tests from CryptTest (0 ms total)

[----------] 4 tests from DataPathTest
[ RUN      ] DataPathTest.Initialize
[       OK ] DataPathTest.Initialize (2 ms)
[ RUN      ] DataPathTest.InitializeInvalid
[       OK ] DataPathTest.InitializeInvalid (0 ms)
[ RUN      ] DataPathTest.UdpBind
[       OK ] DataPathTest.UdpBind (1 ms)
[ RUN      ] DataPathTest.UdpRebind
[       OK ] DataPathTest.UdpRebind (1 ms)
[----------] 4 tests from DataPathTest (5 ms total)

[----------] 1 test from PlatformTest
[ RUN      ] PlatformTest.QuicAddrParsing
[       OK ] PlatformTest.QuicAddrParsing (0 ms)
[----------] 1 test from PlatformTest (0 ms total)

[----------] 37 tests from TlsTest
/root/msquic/src/platform/unittest/TlsTest.cpp:166: Failure
Expected: (nullptr) != (PfxPath), actual: (nullptr) vs NULL
[ RUN      ] TlsTest.Initialize
[       OK ] TlsTest.Initialize (2 ms)
[ RUN      ] TlsTest.Handshake
[       OK ] TlsTest.Handshake (5 ms)
[ RUN      ] TlsTest.HandshakeCertFromFile
Segmentation fault (core dumped)

--> If this also happens for you (?) one probably has to conclude that [msquic](https://github.com/microsoft/msquic is not really well supported/tested under Linux (not totally unexpected as per this statement "Please note that since using CMake directly is not the recommended way of building MsQuic, it's likely that these instructions may fall out of date more often than the Building with PowerShell ones.").

I'm not sure what to do next: Drop msquic or give the PowerShell (under Linux?!) a try....

[igorbarshteyn]

Hi Michael,

I tried running the MSQUIC build (with default quictls for OpenSSL) using PowerShell. Unfortunately, it completely blew up my test VM, with libatomic giving dependency timeouts at around 98% of the way into the build and irrecoverably hanging my system. Granted, I have limited hardware to work with here, but still it's somewhat discouraging.

Another data point: I attempted to build MSQUIC using PowerShell with oqs-openssl-quic (swapping it in for the openssl submodule), and hit the exact same build dependency errors for oqs.h that I had in my CMake attempt. I found that all the PowerShell script does is invoke CMake with the same parameters as the direct CMake build - there was no noticeable difference and errors came up at the same point in the build. It is beyond my ken to dig through Microsoft's convoluted build scripts to ferret out where the missing oqs references are.

I'm going to put down MSQUIC for the time being, and refocus for now on 2 lines of effort:

1) Making a script to automate adding QUIC support to OQS-OpenSSL 1.1.1x - basically automating the git cherry-picks I did and build oqs-openssl-quic at runtime on the client machine, instead of pulling the (now outdated) version from this repository. I've been doing some research on how to go about this and have a couple of ideas in mind.

2) Seeing if I can get LSQUIC to build by making less changes in build.make and associated txt files for failing components. I've been learning more about Make and CMake, so I'm thinking it may be possible to properly define references to liboqs.a in either the configure script or primary Makefile for LSQUIC. This may allow me to automate this and insert the automation into the published oqs-openssl-quic.sh script, instead of resorting to a ZIP of the pre-built LSQUIC. It would allow more people to tinker with the LSQUIC client and offer more transparency.

Should I see success along these 2 directions, I will report back and post in this issue (for the client) or in the other issue you opened regarding automatic QUIC support addition.

Thanks,

Igor

[nibanks]

MsQuic can definitely be built purely via cmake, and Linux is 100% supported. If you have specific questions, feel free to post on our Discussions page. We're happy to help. Discord is also available if you prefer.

[thhous-msft]

The oqs issues in msquic are because of how we interface with the openssl build. Since its not a native cmake build, we have to wrap the perl build, and then we end up with a name dependency on libssl.a. Its actually pretty easy to fix this.

https://github.com/microsoft/msquic/blob/main/submodules/CMakeLists.txt

In there is all the code that handles the build wrapping. The way to fix it is lines 171 and 172. The name of the libraries pointed to just need to be changed.

You might have to change some include paths in that file as well. But everything would be contained there.

[baentsch]

@nibanks @thhous-msft Thanks very much for your pointers/suggestions. Just created oqs-independent https://github.com/microsoft/msquic/discussions/2334. As that resolves, I'll try the oqs-integration again -- just using cmake though: I'm a poor Unix guy without ever having learned PowerShell :)

[baentsch]

@igorbarshteyn fyi, below's a docker setup successfully building and testing msquic; I'll now go about trying to move oqs-openssl-quic in. Will keep you posted.

FROM ubuntu:focal

ENV TZ=Europe/Zurich
ENV DEBIAN_FRONTEND=noninteractive

WORKDIR /root
RUN apt update && apt install software-properties-common apt-utils -y

RUN apt-add-repository ppa:lttng/stable-2.12 -y && apt update && apt install build-essential cmake liblttng-ust-dev lttng-tools git ninja-build libssl-dev wget -y

# Install powershell as testing requires it:
RUN wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb && dpkg -i packages-microsoft-prod.deb && apt-get update && add-apt-repository universe && apt-get install -y powershell

RUN git clone https://github.com/microsoft/msquic && cd msquic && git submodule init && git submodule update --recursive

# builds but test will fail b/o powershell dependency of tests:
# RUN cd /root/msquic && mkdir build && cd build && cmake -G 'Unix Makefiles' ..  && cmake --build .

# testing requires powershell so this won't work:
#RUN /root/msquic/build/bin/Release/msquicplatformtest

# so, let's build using powershell:
RUN pwsh /root/msquic/scripts/build.ps1 -Config Debug -Arch x64 -Tls openssl

# Note: ipv6 must be enabled for all tests to pass OK
#RUN pwsh /root/msquic/scripts/test.ps1

[igorbarshteyn]

@baentsch That's excellent news! If you need to get a "fresh" build made of oqs-openssl-quic, I've added a script that automatically builds it from current OQS-OpenSSL and OpenSSL 1.1.1m+quic (quictls). That script is in a ZIP file I put in a response to Issue #5

Thanks!

--Igor

[baentsch]

@igorbarshteyn FYI, I managed to get msquic to build (again for simple reproducibility in a Docker image) with OQS support: https://github.com/igorbarshteyn/oqs-openssl-quic/tree/mb-client-server-docker/oqs-scripts/msquic . Next step would be to test this -- eventually also against an OQS-enabled server, so I'll need to learn more about the msquic APIs...

[igorbarshteyn]

@baentsch Great news. Let me see if I can deploy this on my test box with Nginx-QUIC (with OQS) and already installed, and run some tests tonight. I think the command is quicsample per the link to https://github.com/microsoft/msquic/blob/main/src/tools/sample/sample.c that you gave earlier. I'll let you know how far I can get. Thanks!

--Igor

[igorbarshteyn]

@baentsch I tried to do some testing tonight with the MSQUIC docker image you posted. Granted, this was my first time experimenting with Docker, so I probably got something wrong.

I was able to build the image, start the container in host network mode on my test VM (which I also set up to run the nginx-quic server I built against oqs-openssl-quic, in ssl_ecdh_curve "auto" mode for maximum compatibility, and listening on localhost/127.0.0.1 port 8443). I then tried to run the MSQUIC quicsample client from the docker container to 127.0.0.1 for port 8443. I used an RSA-2048 cert on the server and the corresponding CA cert with the client on the container (not that it made a difference since I never made it to certificate validation stage). No luck, unfortunately, but it's probably because I'm doing something wrong with Docker networking.

My packet capture result (and commands I ran) are enclosed in the attached ZIP file.

What I'm seeing is that the host receives the initial QUIC packet from the container (which, by the way, specifies classical rather than quantum-safe groups). However the host can't seem to successfully send a packet back to the container (it can't reach the port that the container sends from and gives an ICMP failure message). Interestingly, the sending port on the container seemed to change on each iteration of the test that I ran.

Hoping that you had better luck.

Thanks,

Igor

msquic-testing.zip

[baentsch]

@igorbarshteyn: Thanks for giving it a go. Unfortunately, I can't fully follow the script as I'm at a loss what "recert.sh" is. Also, I don't find a value "ssl_ecdh_curve parameter" to change in the nginx config file /usr/local/nginx/conf/nginx.conf in the docker image I build via docker build -t oqsquicnginx -f Dockerfile-server ..

BTW, as a suggestion here's a shorter way to build and run the docker image:

docker build -t msquic-oqs .
docker run --network host -it msquic-oqs

-> This delivers you straight to a bash in the container where you can then start the quicksample program. Generally, though, it'd be better to run the nginx server image in its container connecting to a dedicated quic network, not the host's network; the names given to the containers can be used within the containers as "host names":

docker network create quic-test
docker run --name nginx --rm --network quic-test -ti oqsquicnginx
docker run --network quic-test -it msquic-oqs

Then you can run msquic in the latter container as msquic/artifacts/bin/linux/x64_Debug_openssl/quicsample -client -cert CA.crt -target:nginx -port:8443

Now all that said, unfortunately all I get is

root@bfaf8fb62414:~# msquic/artifacts/bin/linux/x64_Debug_openssl/quicsample -client -cert CA.crt -target:nginx -port:8443
[conn][0x55dc3db74c70] Connecting...
[conn][0x55dc3db74c70] Shut down by transport, 0x71
[conn][0x55dc3db74c70] All done

--> Same error for you? Probably not, as my nginx image doesn't even have port 8443 open -- so I'm missing something (?)

[nibanks]

What are you trying to achieve by using the MsQuic sample? The sample is simply meant to show you how to use the API. It uses a fake sample ALPN (see here) to negotiate with other sample apps. If you're trying to connect to a real app, it will need to use the appropriate ALPN and protocol on top of QUIC most likely; even if you're just trying to connect at the QUIC layer and do a handshake.

[nibanks]

Oh, and if you're using the MsQuic sample to connect to a server that doesn't have a trusted certificate, you'll probably need to use the -unsecure arg.

[baentsch]

@nibanks Thanks for pointing out what the sample app is for (and what it isn't). I honestly didn't yet look into it. FYI, what I'd like to initially achieve is execute a TLS handshake with a QUIC-enabled server (say nginx) using a classic crypto algorithm and once that succeeds, switch to a (set of) quantum-safe algorithm(s) to see how (whether :) QUIC-TLS handles those algorithms. I also didn't yet look into the msquic APIs to check how to change the algorithms to be used during session setup, so any pointer would be gratefully welcome(d). But then again, could the sample app also be used for this purpose (acting both as QUIC client and server and allowing us to alternate QUIC-TLS setup crypto algorithms)?

[igorbarshteyn]

@baentsch Apologies for lack of clarity. Please allow me to clarify:

I deployed your MSQUIC docker image into the test VM that was configured using the original deployment script for oqs-openssl-quic published on LinkedIn (which, in addition to pulling oqs-openssl-quic from this repo and building it against liboqs, also builds nginx-quic against oqs-openssl-quic, and deploys the LSQUIC client built against oqs-boringssl, as well as wireshark for testing). The tail end of the script also deploys an accessory script called recert.sh (which is the script I use to regenerate certificates for testing).

Regarding the nginx.conf file (a copy of it is also deployed by the above master script), the relevant parameter "ssl_ecdh_curve" is found on line 34. It tells nginx which KEX to use for handshakes. Coupled with the "ssl_prefer_server_ciphers" parameter in line 33 set to "on", it forces the server and client to negotiate using whatever KEX/s is/are set here. This can be set to a single value e.g. "p256_bikel1", a set of values for the client and server to choose from, or to "auto" - which essentially allows client and server to negotiate whatever is common between the two, and this usually results in an X25519 handshake.

I got the same errors as you in my testing (which resulted in the packet capture I enclosed) - shut down by transport. Now that @nibanks has clarified what the quicsample software is and what it's not, the situation is more clear. It's not a full-fledged client for quic/http3 like, for example LSQUIC (which is what I originally tested with).

Thanks both, and I hope this makes my previous post more clear.

Regards,

Igor

[baentsch]

For feedback, here's a "sneak" preview of what's possible using QUIC-OQS-OpenSSL, nginx and msquic now:

• https://hub.docker.com/repository/docker/openquantumsafe/nginxquic
• https://hub.docker.com/repository/docker/openquantumsafe/msquic-reach

[nibanks]

Can you provide any details about what is working exactly? You have post-quantum crypto working with MsQuic?

[baentsch]

Allow me to quote myself (from the page above): "this image integrates quantum safe cryptography (QSC) into the msquic software package to allow exercising all QSC algorithm combinations currently supported by the OpenQuantumSafe project."

So the answer is somewhere between "Yes" and "It depends": The one program this exercises a lot (with all pure quantum-safe and all hybrid classic/QSC algorithms) in interaction with nginx is "reach". You're in a better position to judge what other components within msquic then should work quantum-safe.

Edit/Add: Upon re-reading, just to clarify: My answer above is not meant to be deregatory -- if so only with regard to my understanding of (MS)QUIC: @igorbarshteyn worked for quite some time on the topic, both finding quictls and a way to cherrypick the nicely separated changes to upstream openssl there onto our oqs-openssl fork. I only dug --with your help-- into msquic and its build process in order to bring liboqs to "play nice" (and then to exercise all algorithms in a way to give meaningful results wrt the suitability of all algorithm combinations in QUIC). Thus, again, I wouldn't go so far as to say this ensures "post-quantum crypto working with MsQuic" -- I just understand too little of QUIC to make such broad statement -- hence my (implied) question to you to state what (test suite?) you'd wants to see working to give a clear positive answer to your question.

[nibanks]

My answer above is not meant to be deregatory

No offense taken. 😄

my (implied) question to you to state what (test suite?) you'd wants to see working to give a clear positive answer to your question.

At this point, I am just interested in knowing that the QUIC handshakes seem to succeed, and with what algorithms they work. We can follow up on more interesting test cases (ones with random packet loss during the handshake come to mind) at a later date.

[igorbarshteyn]

@baentsch Thanks so much for putting those docker images together - I can't wait to do some testing work when I have more cycles to work on this project after the 7th! (and also to review and test your amended auto-build script for oqs-openssl-quic - I read through it and it looks very impressive but I haven't tried it out yet)

@nibanks While @baentsch has graciously taken on the effort of exploring MSQUIC as a client back end, I did manage to run some early tests with LSQUIC and nginx-quic as part of my initial demo, and was able to get some results with quantum-safe handshakes over QUIC, showing that they do work in some cases albeit with fragmented packets. You can find my preliminary test data in https://github.com/igorbarshteyn/oqs-openssl-quic/issues/1#issuecomment-1016538488 (I enclosed a couple of packet capture archives and a summary Excel table). Much room remains for improvement, and MSQUIC may be a better client stack. I hope to work with the team further on this project after my return next week :)

Have a good evening everyone,

--Igor

[baentsch]

@igorbarshteyn The big advantage of using msquic is that it's using openssl (as opposed to 'boringssl') so it inherits the more complete set of quantum-safe algorithm combinations (incl. all signature hybrids). That said, using the data transfer statistics built into msquic one first result is that it seems that if the pure quantum-safe algorithm works in QUIC, the hybrid will also work: Here's a brief excerpt of a run evaluating all 3000+ algorithm combinations:

QSC signature algorithm, QSC KEM algorithm, conn established, #UDP packets received, #UDP packets sent, #bytes received
[...]
picnicl1full,frodo640aes,0,40,12,40260
picnicl1full,frodo640shake,0,40,12,40260
picnicl1full,frodo976aes,0,53,19,50312
picnicl1full,frodo976shake,0,53,18,50023
picnicl1full,frodo1344aes,0,63,20,56487
picnicl1full,frodo1344shake,0,63,24,56555
picnicl1full,kyber512,0,28,8,33861
picnicl1full,kyber768,0,29,8,34433
picnicl1full,kyber1024,0,29,4,34625
picnicl1full,ntru_hps2048509,0,28,7,33911
picnicl1full,ntru_hps2048677,0,29,7,33988
picnicl1full,ntru_hps4096821,0,29,4,34290
picnicl1full,ntru_hps40961229,0,30,4,35259
picnicl1full,ntru_hrss701,0,29,4,34214
picnicl1full,ntru_hrss1373,0,31,8,35731
picnicl1full,lightsaber,0,28,7,33914
[...]
 dilithium5,p256_frodo640aes,1,23,13,18957
dilithium5,p256_frodo640shake,1,23,11,18957
dilithium5,p384_frodo976aes,1,33,19,25527
dilithium5,p384_frodo976shake,1,33,17,25527
dilithium5,p521_frodo1344aes,0,43,21,31977
dilithium5,p521_frodo1344shake,0,43,25,31977
dilithium5,p256_kyber512,1,8,3,9232
dilithium5,p384_kyber768,1,9,4,9635
dilithium5,p521_kyber1024,1,10,4,10203
dilithium5,p256_ntru_hps2048509,1,8,3,9163
dilithium5,p384_ntru_hps2048677,1,9,4,9475
dilithium5,p521_ntru_hps4096821,1,9,4,9814
dilithium5,p521_ntru_hps40961229,1,10,4,10477
dilithium5,p384_ntru_hrss701,1,9,4,9686
dilithium5,p521_ntru_hrss1373,1,12,5,11138
dilithium5,p256_lightsaber,1,8,3,9200
dilithium5,p384_saber,1,9,4,9635
dilithium5,p521_firesaber,1,10,4,10107
[...]
p521_dilithium5,p256_frodo640aes,1,24,13,19281
p521_dilithium5,p256_frodo640shake,1,24,11,19279
p521_dilithium5,p384_frodo976aes,1,34,17,25852
p521_dilithium5,p384_frodo976shake,1,34,17,25853
p521_dilithium5,p521_frodo1344aes,0,44,24,32301
p521_dilithium5,p521_frodo1344shake,0,44,24,32301
p521_dilithium5,p256_kyber512,1,8,3,9506
p521_dilithium5,p384_kyber768,1,10,4,9959
p521_dilithium5,p521_kyber1024,1,10,4,10477
p521_dilithium5,p256_ntru_hps2048509,1,8,3,9438
p521_dilithium5,p384_ntru_hps2048677,1,9,4,9750
p521_dilithium5,p521_ntru_hps4096821,1,10,4,10139
p521_dilithium5,p521_ntru_hps40961229,1,10,4,10751
p521_dilithium5,p384_ntru_hrss701,1,10,4,10010
p521_dilithium5,p521_ntru_hrss1373,1,12,5,11413
p521_dilithium5,p256_lightsaber,1,8,3,9474
[...]
p256_picnicl1full,frodo640aes,0,43,13,43697
p256_picnicl1full,frodo640shake,0,37,11,36600
p256_picnicl1full,frodo976aes,0,53,18,50390
p256_picnicl1full,frodo976shake,0,53,17,50526
p256_picnicl1full,frodo1344aes,0,63,23,56716
p256_picnicl1full,frodo1344shake,0,63,23,56854
p256_picnicl1full,kyber512,0,28,9,34057
p256_picnicl1full,kyber768,0,29,7,34425
p256_picnicl1full,kyber1024,0,30,6,34974
p256_picnicl1full,ntru_hps2048509,0,28,3,34191
p256_picnicl1full,ntru_hps2048677,0,29,8,34406
p256_picnicl1full,ntru_hps4096821,0,29,6,34470
p256_picnicl1full,ntru_hps40961229,0,30,5,35030
p256_picnicl1full,ntru_hrss701,0,29,5,34274
p256_picnicl1full,ntru_hrss1373,0,31,7,35926
p256_picnicl1full,lightsaber,0,28,8,33889
[...]

(a "0" in the third column means no connection could be established; the other columns indicate number of packets/fragmentation).

--> One can see immediately the advantages of (even NIST level 5) dilithium over (level 1) picnic and the drawbacks of frodo1344...

Which other QUIC "connection quality metrics" would you consider interesting to judge the quality of QSC algorithms? Particulary @nibanks: Which metrics have you built into msquic and would you like to see exposed in such "comparative tests"? Another question: How do you evaluate QUIC in the presence of "bad networks"? Is there a way to configure msquic to simulate network losses? Is there a way already built-in to get timing statistics?

[baentsch]

Actually, question to both of you: Would you be interested in posting such comparative execution results, e.g., to inform the NIST PQC competition? Maybe even do this jointly as this is at the "crossroads" of our respective areas of interest?

[nibanks]

Which metrics have you built into msquic and would you like to see exposed in such "comparative tests"?

These handshake timing values would be useful. The units are microseconds. It would give you a comparative time cost doing the handshake.

Is there a way to configure msquic to simulate network losses?

In debug builds we expose test hooks that can be used for simulating packet loss. Our test code does exactly this. I can help you more on this if you're interested.

Would you be interested in posting such comparative execution results

Most likely. What exactly do you have in mind, beyond what you've already done above?

[nibanks]

Looking more closely at the results above, it seems that the only cases that succeed are where the bytes received are less than 20,000. My initial guess is that there is some limiting buffer involved here. I know on MsQuic we default fairly small buffers for send at the TLS layer.

Server: 8KB Client: 4KB

These values are controlled here by QUIC_MAX_TLS_CLIENT_SEND_BUFFER and QUIC_MAX_TLS_SERVER_SEND_BUFFER. We also cap the max receive buffer to QUIC_DEFAULT_STREAM_FC_WINDOW_SIZE / 2 which is effectively 16KB.

I'm willing to bet one or more of those caps would be interfering with your tests. the numbers were picked based on casual examination of existing TLS usage. I'm sure they can be improved.

[baentsch]

These handshake timing values would be useful. The units are microseconds. It would give you a comparative time cost doing the handshake.

Those would indeed be very interesting. When having added the corresponding code (correctly I hope), the values don't really make sense, though:

root@14e8c8ccc652:~/msquic# quicreach -server:quic.nginx.org -alpn:h3

quic.nginx.org:443:

QUIC_STATISTICS Start: 320708632560
QUIC_STATISTICS InitialFlightEnd: 0
QUIC_STATISTICS HandshakeFlightEnd: 0
QUIC_STATISTICS Handshake Bytes: 0
  0x00000001           h3    reachable

FYI I added this code to reach.cpp:

diff --git a/src/tools/reach/reach.cpp b/src/tools/reach/reach.cpp
index b895c3e8..fb37ea72 100644
--- a/src/tools/reach/reach.cpp
+++ b/src/tools/reach/reach.cpp
@@ -115,6 +115,25 @@ CXPLAT_THREAD_CALLBACK(TestReachability, _Alpn)
         exit(1);
     }

+    QUIC_STATISTICS value = {};
+    uint32_t valueSize = sizeof(value);
+    QUIC_STATUS Status =
+        MsQuic->GetParam(
+            Connection,
+            QUIC_PARAM_LEVEL_CONNECTION,
+            QUIC_PARAM_CONN_STATISTICS,
+            &valueSize,
+            &value);
+    if (QUIC_FAILED(Status)) {
+        printf("MsQuic->GetParam(CONN_STATISTICS) failed, 0x%x.\n", Status);
+    }
+    else {
+        printf("QUIC_STATISTICS Start: %ld\n", value.Timing.Start);
+        printf("QUIC_STATISTICS InitialFlightEnd: %ld\n", value.Timing.InitialFlightEnd);
+        printf("QUIC_STATISTICS HandshakeFlightEnd: %ld\n", value.Timing.HandshakeFlightEnd);
+       printf("QUIC_STATISTICS Handshake Bytes: %d\n", value.Handshake.ClientFlight1Bytes);
+    }
+
     MsQuic->ConfigurationClose(Configuration);
     CxPlatEventWaitForever(Context.Complete);
     CxPlatEventUninitialize(Context.Complete);

-> Would you have an idea what's going on here?

What exactly do you have in mind, beyond what you've already done above?

Timings and error rates (retransmissions) would be interesting as they could paint a picture as to which algorithms are both fast and small enough to be good classic crypto replacements. That said, are you aware of QUIC publications (and/or measurements already covering this topic (i.e., which --presumably classic-- crypto works best for QUIC handshaking)?

[baentsch]

-> Would you have an idea what's going on here?

Extending my own question after reading through msquic code: These counters seem to get set when cert validation is done, right? That ("QuicCryptoCustomCertValidationComplete") in turn doesn't seem to be called -- ever. Same statement holds true for "QuicCryptoProcessTlsCompletion": Never ever called when executing running against our own nginx: How can this be (it has its own root CA that is set via the openssl SSL_CERT_FILE env var)? When running against quic.nginx.org that function does get called (why only then?) but none of the Handshake data structures (other than Timing.Start) get populated. Puzzling. Worthwhile opening a discussion in msquic for this topic?

[nibanks]

FYI I added this code to reach.cpp:

That's too early. You need to query the data after the connection succeeds, around here.

[baentsch]

That's too early. You need to query the data after the connection succeeds, around here.

That's where I started, but this always returns QUIC_FAILED at the GetParam call -- which I found kind of possible as the connection structure may have been destroyed at that time (didn't look into what "CxPlatEventUninitialize" exactly does). Anyway: Moved the code there, results in:

  0x00000001           h3    reachable
MsQuic->GetParam(CONN_STATISTICS) failed, 0x1.

Explanations for this result code I didn't find in the documentation. Any pointers very welcome.

[nibanks]

Ah, you're right. The connection is being closed in the callback, so you need to query in the callback, here.

[baentsch]

so you need to query in the callback, here.

Ahhh -- right: Thanks! I don't want to output data in a callback, so lazy as I was it never occurred to me to extend the Context structure and pass out the handshake values that way for "outside-of-callback" printing -- or is that not necessary (i.e., are the Callback functions thread safe and not timing critical)?

[nibanks]

are the Callback functions thread safe and not timing critical

They are generally critical, so ideally you wouldn't do the printf in the callback; but you can query the statistics in the callback without any fear of timing issues. Then print them on the main thread.

[baentsch]

Closed via https://github.com/open-quantum-safe/oqs-demos/pull/123 and https://github.com/open-quantum-safe/oqs-demos/pull/124