Open sftcd opened 1 year ago
sftcd
commented
1 year ago Just checked using wireshark, and it looks like the same issue arises when accessing https://tls-ech.dev which is another ECH test server operated by someone else and not using my code. That server seems to only support x25519 so also triggers HRR when the client initially chooses p256.
jpbland1
commented
12 months ago Hey @sftcd,
I've refactored the ECH acceptance code to handle writing and checking confirmation bytes for a HelloRetryRequest. I wasn't able to recreate the error you described, when I ran your example command pre-update I didn't get the same error, it's possible I needed to do something else to trigger an HRR? If so let me know and if you are able to re-test with the new code I would appreciate it, thanks again for bringing these bugs to our attention.
Best Wishes, John Bland
sftcd
commented
12 months ago I'll take a look at it again this evening. Thanks for being so responsive too btw!
sftcd
commented
12 months ago Hiya,
I tried that out locally, but it didn't work for HRR first time out. Probably no surprise if you've not been able to test much. I'll look some more in a bit, but to trigger HRR, you can either use curl with one of my servers or the one at tls-ech.dev.
The server hosting https://draft-13.esni.defo.ie:8414/stats is an openssl s_server instance that's configured to only support the p384 group, so if you access that with almost any client (that doesn't send an initial ClientHello with a p384 key-share), then that'll trigger HRR. You can verify that with wireshark, where you'll see the ClientHello causing a HRR response and then the 2nd ClientHello.
It looks to me like the server at tls-ech.dev also triggers HRR for the curl+wolf client, since that server only appears to support x25519, and curl+wold only sends a p256 key share in the initial ClientHello. (Note that there was a 2nd issue with wolfSSL and that server though, not sure if that's fixed in your PR.)
I'll look some more see what I can see...
S.
sftcd
commented
12 months ago I had a look at that in wireshark and vs. a localhost server and unless I've gotten my build wrong, it looks like that version is still setting the enc field in the 2nd ClientHello. (And the values differ which may indicate not re-using the same HPKE context?)
One thing that looked odd to me was this code - I'd have expected the msgType would be client_hello when sending the 2nd ClientHello and not hello_retry_request but maybe I'm misunderstanding what's going on there?
sftcd
commented
12 months ago BTW @jpbland1 if it's useful to do a call to chat about this just ping me by email and we can find a slot
jpbland1
commented
12 months ago Hey @sftcd,
I've made fixes for hrr and have it working from wolfssl->wolssl. You were right about failing to write the empty string for enc in the second client hello, I needed to check the hpke context to see if it was setup instead of trying to look at the msgType, which made no sense. There were other issues with how I was computing the transcript hash but I think I have that fixed. When I run your example now I get:
$ ./src/curl --ech https://tls-ech.dev/
curl: (77) CA signer not available for verificationI'm still looking through it so I don't know if this is because of ech failure. ECH with HRR is working wolfssl->wolfssl so if there's something I'm doing wrong for ECH I'm doing it wrong consistently between client and server.
Best Wishes, John Bland
sftcd
commented
12 months ago Hiya,
Still the same branch, right?
I pulled that and tried it, but still not working so far. (ECH+HRR is a real pain:-)
In wireshark I still see an enc value (different from that in the 1st ClientHello) being sent to tls-ech.dev in the 2nd ClientHello and the same with my localhost test, and with different enc values in the first and second ClientHello messages. What do you see with wireshark with your wolff->wolf testing?
From the 2nd CH sent to tls-ech.dev, wireshark shows me this ECH extension value:
0000 00 00 01 00 01 2b 00 20 5c 15 51 78 ca 2c 81 d2 .....+. \.Qx.,..
0010 57 b0 d9 6f 5d 68 a8 5c 4d 7e be 9e 26 5c 80 da W..o]h.\M~..&\..
0020 4a 4e 5e 13 e5 fc bc 4f 01 10 be 5e 3c 3d 0c cb JN^....O...^<=..
0030 53 2f 9e a7 4a ff 7e b2 69 13 1e 56 af 1c a0 1a S/..J.~.i..V....
0040 6c 49 fa 18 53 4a 18 fb 88 10 d9 73 f1 fc 24 3e lI..SJ.....s..$>
0050 56 2a 61 5c 8a df d9 45 cb ac f7 aa 06 aa bb 31 V*a\...E.......1
0060 9a c6 c3 4a ca b2 78 cc d8 2e 75 ae aa e5 a8 2d ...J..x...u....-
0070 2b 09 f7 4d 17 7d 14 ff 75 24 6d b5 12 f1 15 e6 +..M.}..u$m.....
0080 39 69 74 28 aa 03 c0 2a 44 d1 ab d4 ab 45 12 b7 9it(...*D....E..
0090 6c 27 af d1 fd 72 15 86 11 b3 b9 c9 35 78 e4 62 l'...r......5x.b
00a0 b7 69 8c 5f 64 9e 4f e0 82 ac cb 7d 01 b2 ad 73 .i._d.O....}...s
00b0 66 cb 5d d7 8a 4a 46 b3 8d 59 d2 14 f1 44 72 8b f.]..JF..Y...Dr.
00c0 19 74 ef f7 fe 59 de 02 21 c4 a2 6b a6 97 34 67 .t...Y..!..k..4g
00d0 81 fe d5 77 6a 91 fb 93 a5 b7 d6 f1 f6 2f 1e c9 ...wj......../..
00e0 a3 e8 0d 0c 76 bd cc a2 1e f4 ab 61 04 c9 e5 2e ....v......a....
00f0 fa 3a eb 0e 9f 1f cf 34 01 7b d5 f9 e2 3f 28 fa .:.....4.{...?(.
0100 57 26 de d1 b2 76 89 ea 7f 8f e5 dd 93 77 a1 d7 W&...v.......w..
0110 a7 55 7e 33 1f 5b 0f 4a a9 63 cc ad 35 dc 99 64 .U~3.[.J.c..5..d
0120 3a 0e 88 1b 45 c0 c1 bc d0 27 b1 c3 16 34 90 b3 :...E....'...4..
0130 d7 cf 79 d4 76 03 17 4c 56 77 ..y.v..LVwThe 0x00 0x20 in octets 7 & 8 is the giveaway - that's the lenght for a 32 octet enc value.
In my localhost test trace, I parse out the ECH from the 2nd ClientHello and see similarly:
extension_type=encrypted_client_hello(draft-13)(65037), length=410
ECH-type is outer
kdf 1, aead: 1, config_id: 0xbb
enc (len=32): C44A661AF9650BE5126DC345134E29C38D266A926E9E365EBDAFA1D7B0A40F38
payload (len=368): 211C1EC5E94276A8815AD5F3CEB8AD1E6B1AB41F0E5C6B0C3EBE8ECF434062CBCA26CD40CB9E748188E88FE7393286B4CD026000BFA6849FD64F385DDFA7B2EBD78C50A603A1C7A391CA9D7423861D4A0459D914E0D8E2A4CD93FD251309022F01F9B42FFCA31126E45400A800A7251969331632F55DB0110534DA170D7C43087573BE78E56DA6120EC642B42FA0C141BB6166EF641C424CF00D65A4F9FBD821CAA4793BF195FE66DE420B2FB5FA751E2E44D25DE1E937925121185179B8DB5D214A42DC47B43B995AA7B6B553EF58A0DD7FD7C27C53CD986154D79D3AAA3E821BE6CABB5B99FD4552CD94BFD65CD23EF67C7854C652EEE2EA77C0015E36FE584D1E6EDD1EC4511FDE4123ADB41C00302D07113CEC755346768BD89C8A5C814251AFB6492BD4452A83D144D833455E368C6FC39BAE5318D36AD5FE7DAE9FB9C4CBA690641BAD5D9D030ED1C4EC814579042FB3614CAFB9BDC438A0A6B16268229100955E092B3D3F12699650B9BCA9A9BTW, I also see a double-free crash at the end of talking to tls-ech.dev here.
...
* ALPN: curl offers http/1.1
* ECH: ECHConfig from DoH HTTPS RR
* ECH: imported ECHConfigList of length 75
* SSL_connect failed with error -313: received alert fatal error
* Closing connection
double free or corruption (!prev)
Aborted (core dumped)If you're not seeing that, I can try see where it's coming from.
sftcd
commented
12 months ago Also forgot to say: I need to run curl+wolf in insecure mode to get it to work with most of my servers - not sure why that is, but curl+openssl is happy with the certs, so adding a -k to your curl command line would likely sort the cert error you showed above. I believe (though not 100% sure) that's a known difference of opinions between wolfSSL and OpenSSL devs and not a bug and should go away if I configure my servers differently. The -k isn't relevant to ECH processing at all though, so is fine for this testing.
jpbland1
commented
12 months ago Hey Stephen @sftcd,
I found the issue and made another commit, when wolfssl tries to create all the extensions for a clienthello it was over-writing the old ech extension with a new one, destroying the hpke context. I've updated it and have confirmed in gdb that it's sending the zeros.
I'm trying to do your curl test but I'm still getting grease only, when I run it in gdb it looks like the grease ech happens successfully and it fills the config but from what I can see in gdb it doesn't look like there is a second connection attempt being made after the grease succeeds, I've added some comments to this output with #:
$ ./src/curl --ech https://draft-13.esni.defo.ie:8414/stats -k
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/john/Documents/curl/src/.libs/curl...
(No debugging symbols found in /home/john/Documents/curl/src/.libs/curl)
# Break on ech parse so we can see what kind of ech messages the server sends
(gdb) b TLSX_ECH_Parse
Function "TLSX_ECH_Parse" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (TLSX_ECH_Parse) pending.
(gdb) r
Starting program: /home/john/Documents/curl/src/.libs/curl --ech https://draft-13.esni.defo.ie:8414/stats -k
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff67ff640 (LWP 136112)]
[Thread 0x7ffff67ff640 (LWP 136112) exited]
# The first message is from an HRR, 8 bytes long but we don't use it since we're still grease
Thread 1 "curl" hit Breakpoint 1, TLSX_ECH_Parse (ssl=0x5555555dbd50, readBuf=0x5555555d3c0c "\241\a\322\v@\375\350y", size=8, msgType=6 '\006') at src/tls.c:11726
11726 int ret = 0;
(gdb) n
11732 byte* readBuf_p = (byte*)readBuf;
(gdb)
11733 WOLFSSL_MSG("TLSX_ECH_Parse");
(gdb)
11734 if (size == 0)
(gdb)
11737 if (msgType == encrypted_extensions) {
(gdb)
11744 else if (msgType == hello_retry_request && ssl->options.useEch) {
(gdb)
11755 else if (msgType == client_hello && ssl->ctx->echConfigs != NULL) {
(gdb)
11855 return ret;
(gdb) c
Continuing.
# Second message is encrypted extensions, set the ech config for next time
Thread 1 "curl" hit Breakpoint 1, TLSX_ECH_Parse (ssl=0x5555555dbd50, readBuf=0x5555555c2fbe "\001", size=258, msgType=8 '\b') at src/tls.c:11726
11726 int ret = 0;
(gdb) n
11732 byte* readBuf_p = (byte*)readBuf;
(gdb)
11733 WOLFSSL_MSG("TLSX_ECH_Parse");
(gdb)
11734 if (size == 0)
(gdb)
11737 if (msgType == encrypted_extensions) {
(gdb)
11738 ret = wolfSSL_SetEchConfigs(ssl, readBuf, size);
(gdb)
11740 if (ret == WOLFSSL_SUCCESS)
(gdb)
11741 ret = 0;
(gdb)
11855 return ret;
# After the successful grease ECH it should make another connection with the new config but it doesn't, it never reaches SendTls13ClientHello
(gdb) b SendTls13ClientHello
Breakpoint 2 at 0x7ffff7b9e270: file src/tls13.c, line 6571.
(gdb) c
Continuing.
<HTML><BODY BGCOLOR="#ffffff">
<h1>OpenSSL with ECH</h1>
<h2>
ECH attempt we interpret as GREASE
</h2>
<h2>TLS Session details</h2>
<pre>
<pre>
s_server -quiet -key /home/defo/.acme.sh/esni.defo.ie/esni.defo.ie.key -cert /home/defo/.acme.sh/esni.defo.ie/esni.defo.ie.cer -key2 /home/defo/.acme.sh/esni.defo.ie/esni.defo.ie.key -cert2 /home/defo/.acme.sh/esni.defo.ie/esni.defo.ie.cer -CApath /etc/ssl/certs/ -cert_chain /home/defo/.esni//chain.pem -port 8414 -no_ssl3 -no_tls1 -no_tls1_1 -no_tls1_2 -ech_dir /home/defo/.ech13/echkeydir/cover.defo.ie.443 -servername draft-13.esni.defo.ie -WWW -groups P-384
This TLS version forbids renegotiation.
Ciphers supported in s_server binary
TLSv1.3 :TLS_AES_256_GCM_SHA384 TLSv1.3 :TLS_CHACHA20_POLY1305_SHA256
TLSv1.3 :TLS_AES_128_GCM_SHA256 TLSv1.2 :ECDHE-ECDSA-AES256-GCM-SHA384
TLSv1.2 :ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 :DHE-RSA-AES256-GCM-SHA384
TLSv1.2 :ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 :ECDHE-RSA-CHACHA20-POLY1305
TLSv1.2 :DHE-RSA-CHACHA20-POLY1305 TLSv1.2 :ECDHE-ECDSA-AES128-GCM-SHA256
TLSv1.2 :ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 :DHE-RSA-AES128-GCM-SHA256
TLSv1.2 :ECDHE-ECDSA-AES256-SHA384 TLSv1.2 :ECDHE-RSA-AES256-SHA384
TLSv1.2 :DHE-RSA-AES256-SHA256 TLSv1.2 :ECDHE-ECDSA-AES128-SHA256
TLSv1.2 :ECDHE-RSA-AES128-SHA256 TLSv1.2 :DHE-RSA-AES128-SHA256
TLSv1.0 :ECDHE-ECDSA-AES256-SHA TLSv1.0 :ECDHE-RSA-AES256-SHA
SSLv3 :DHE-RSA-AES256-SHA TLSv1.0 :ECDHE-ECDSA-AES128-SHA
TLSv1.0 :ECDHE-RSA-AES128-SHA SSLv3 :DHE-RSA-AES128-SHA
TLSv1.2 :RSA-PSK-AES256-GCM-SHA384 TLSv1.2 :DHE-PSK-AES256-GCM-SHA384
TLSv1.2 :RSA-PSK-CHACHA20-POLY1305 TLSv1.2 :DHE-PSK-CHACHA20-POLY1305
TLSv1.2 :ECDHE-PSK-CHACHA20-POLY1305 TLSv1.2 :AES256-GCM-SHA384
TLSv1.2 :PSK-AES256-GCM-SHA384 TLSv1.2 :PSK-CHACHA20-POLY1305
TLSv1.2 :RSA-PSK-AES128-GCM-SHA256 TLSv1.2 :DHE-PSK-AES128-GCM-SHA256
TLSv1.2 :AES128-GCM-SHA256 TLSv1.2 :PSK-AES128-GCM-SHA256
TLSv1.2 :AES256-SHA256 TLSv1.2 :AES128-SHA256
TLSv1.0 :ECDHE-PSK-AES256-CBC-SHA384 TLSv1.0 :ECDHE-PSK-AES256-CBC-SHA
SSLv3 :SRP-RSA-AES-256-CBC-SHA SSLv3 :SRP-AES-256-CBC-SHA
TLSv1.0 :RSA-PSK-AES256-CBC-SHA384 TLSv1.0 :DHE-PSK-AES256-CBC-SHA384
SSLv3 :RSA-PSK-AES256-CBC-SHA SSLv3 :DHE-PSK-AES256-CBC-SHA
SSLv3 :AES256-SHA TLSv1.0 :PSK-AES256-CBC-SHA384
SSLv3 :PSK-AES256-CBC-SHA TLSv1.0 :ECDHE-PSK-AES128-CBC-SHA256
TLSv1.0 :ECDHE-PSK-AES128-CBC-SHA SSLv3 :SRP-RSA-AES-128-CBC-SHA
SSLv3 :SRP-AES-128-CBC-SHA TLSv1.0 :RSA-PSK-AES128-CBC-SHA256
TLSv1.0 :DHE-PSK-AES128-CBC-SHA256 SSLv3 :RSA-PSK-AES128-CBC-SHA
SSLv3 :DHE-PSK-AES128-CBC-SHA SSLv3 :AES128-SHA
TLSv1.0 :PSK-AES128-CBC-SHA256 SSLv3 :PSK-AES128-CBC-SHA
---
Ciphers common between both SSL end points:
TLS_AES_128_GCM_SHA256 TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256
ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-CHACHA20-POLY1305 DHE-RSA-CHACHA20-POLY1305
ECDHE-RSA-AES128-SHA256 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-SHA384 ECDHE-ECDSA-AES256-SHA ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256
DHE-RSA-AES128-SHA256 DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA
Signature Algorithms: ECDSA+SHA512:ECDSA+SHA384:ECDSA+SHA256:ECDSA+SHA1:RSA-PSS+SHA512:RSA-PSS+SHA512:RSA-PSS+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA256:RSA-PSS+SHA256:RSA+SHA512:RSA+SHA384:RSA+SHA256:RSA+SHA224:RSA+SHA1
Shared Signature Algorithms: ECDSA+SHA512:ECDSA+SHA384:ECDSA+SHA256:RSA-PSS+SHA512:RSA-PSS+SHA512:RSA-PSS+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA256:RSA-PSS+SHA256:RSA+SHA512:RSA+SHA384:RSA+SHA256:RSA+SHA224
Supported groups: secp521r1:secp384r1:secp256r1:x25519:secp224r1:ffdhe2048
Shared groups: secp384r1
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_128_GCM_SHA256
Session-ID: FE02CE25EF4C0F275A00A028FE4D20AEF41B26D7FF6F0D416D4BCBCC0211387E
Session-ID-ctx: 01000000
Resumption PSK: 4C5F7F69CDA1EFAE8349239EB3B46257C7EFA5EDD8C130EC43D4859851971DCF
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1695766288
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
0 items in the session cache
0 client connects (SSL_connect())
0 client renegotiates (SSL_connect())
0 client connects that finished
0 server accepts (SSL_accept())
0 server renegotiates (SSL_accept())
11 server accepts that finished
0 session cache hits
0 session cache misses
0 session cache timeouts
0 callback cache hits
0 cache full overflows (128 allowed)
---
no client certificate available
</pre></BODY></HTML>
[Inferior 1 (process 136109) exited normally]I also double checked and wolfSSL_SetEchConfigs only get's called once from retry configs so it's not as if it's being set from DNS and is failing. Let me know if I'm missing something, I'm available anytime tomorrow for a call if you want to discuss this.
Best Wishes, John Bland
sftcd
commented
12 months ago I'm available anytime tomorrow for a call if you want to discuss this.
Suspect a call's a good option. Ping me via email (top of the bug report) and we can arrange a time.
S.
Contact Details
github is fine, or stephen.farrell@cs.tcd.ie
Version
cloned from master last week
Description
When doing ECH, if the client receives a HRR back instead of a normal ServerHello, the 2nd ClientHello is supposed to re-use the HPKE context that was used with the first ClientHello, and in that case, to not send the "enc" field of the ECH extension in the 2nd ClientHello. It looks like wolfSSL is maybe instead creating a new HPKE context for the 2nd ClientHello, which is then treated as GREASE by my ECH-enabled server, as attempting decryption with the HPKE context from the 1st ClientHello fails when presented with the ciphertext of the 2nd ClientHello ECH extension.
And yes, HRR is a pain, and ECH+HRR is a bigger pain;-)
I've not yet looked at the wolfSSL code to see what's what but plan to, so don't have any recommendation for a fix yet.
I have tested my ECH+HRR setup with browsers and my own OpenSSL client and it seems to work with those, but it's of course possible there's also some issue(s) on the server side too, we'll see.
Reproduction steps
If you build my experimental cURL fork (HOWTO), then you can test that against a server I've deployed that only supports the P384 curve, which should trigger HRR. So:
I'm not sure that's exactly the right error, but we can look at that separately if needed.
Note that the ECH public key for that server changes hourly, so needs to be fetched from the relevant HTTPS RR in the DNS. (The curl fork does that internally.)
Relevant log output
Second ClientHello: