search

steffengy / schannel-rs

Schannel API-bindings for rust (provides an interface for native SSL/TLS using windows APIs)
MIT License
49 stars 51 forks source link

Strange handshake errors when running multiple TLS servers on the same machine (through rust-native-tls) #95

Closed jfaust closed 1 year ago

jfaust commented 1 year ago

I have a project where I'm generating a self-signed certs with rcgen and then using that cert with rust-native-tls & hyper. I've run into some strange problems when testing that I've isolated into a repro. I believe this is an schannel issue, because I can only reproduce it on Windows, and the handshake error is coming from this crate.

To reproduce (the order of operations here is very important):

  1. Clone https://github.com/jfaust/rust-self-signed-native-tls
  2. In one shell, run cargo run --example server 12345
  3. In another shell, run curl -v --insecure https://localhost:12345/ - it should succeed. This is just to confirm that everything seems to be working. Running that command over and over works just fine.
  4. In another shell, run cargo run --example server 12346
  5. Now run that same curl command again (curl -v --insecure https://localhost:12345/). I get this error: 
    curl -v --insecure https://localhost:12345/
*   Trying 127.0.0.1:12345...
* Connected to localhost (127.0.0.1) port 12345 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* [CONN-0-0][CF-SSL] TLSv1.0 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Unknown (21):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS alert, decrypt error (563):
* OpenSSL/3.0.8: error:02000086:rsa routines::last octet invalid
* Closing connection 0
curl: (35) OpenSSL/3.0.8: error:02000086:rsa routines::last octet invalid

    Sometimes it's last octet invalid, sometimes first octet invalid, sometimes data too large for modulus.

  6. If you restart the first server, the curl command will work again. However, if you now run it against the second server (port 12346), that one no longer works (you can test it before step (5) to confirm it worked initially).

It seems like there is some kind of cross-process corruption happening somehow, and I am baffled as to how that would be the case. As far as I can tell, the two processes are completely disconnected from each other - they're bound to different ports, they use different self-signed certs (generated at process startup), and they're not communicating with each other in any way.

Any thoughts on what could be happening here?

jfaust commented 1 year ago

Of course, as soon as I posted this I had a thought, and it turned out to be correct. The problem is that rust-native-tls is using the same container name for the store no matter what. Not sure what the right solution here is, but I don't think it's a problem in schannel-rs.