search

unitedstates / inspectors-general

Collecting reports from Inspectors General across the US federal government.
https://sunlightfoundation.com/blog/2014/11/07/opengov-voices-opening-up-government-reports-through-teamwork-and-open-data/
Creative Commons Zero v1.0 Universal
107 stars 21 forks source link

[arc] TLS handshake fails #289

Closed divergentdave closed 8 years ago

divergentdave commented 8 years ago

Good news: ARC is now redirecting to HTTPS Bad news: the server hangs up on too-new OpenSSL clients (fails on Ubuntu 16.04's OpenSSL 1.0.2g-1ubuntu4.1, SSL Labs's OpenSSL 1.0.1l, and SSL Lab's OpenSSL 1.0.2e) See also https://www.ssllabs.com/ssltest/analyze.html?d=www.arc.gov. The exception is ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:600)

Looking at packet captures, the server closes the connection right after the Client Hello. The only differences between an OK hello and a bad hello are in cipher suites and elliptic curves. The records in both captures are padded out to 512 bytes, so this isn't another buggy F5 device. I was able to get openssl s_client to connect by adding any of the -tls1, -no_tls1_1, or -no_tls1_2 flags, so a workaround is feasible.

divergentdave commented 8 years ago

With problem:

$ openssl version
OpenSSL 1.0.2g-fips  1 Mar 2016
$ openssl ciphers
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DH-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DH-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DH-RSA-AES256-SHA256:DH-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DH-RSA-AES256-SHA:DH-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:DH-RSA-CAMELLIA256-SHA:DH-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DH-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:DH-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DH-RSA-AES128-SHA256:DH-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DH-RSA-AES128-SHA:DH-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DH-RSA-SEED-SHA:DH-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:DH-RSA-CAMELLIA128-SHA:DH-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DH-RSA-DES-CBC3-SHA:DH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA

Without problem

$ openssl version
OpenSSL 1.0.1t  3 May 2016
$ openssl ciphers
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA
divergentdave commented 8 years ago

Further results: taking the known-good ciphersuite list, removing DES-CBC3-SHA, and using that in openssl s_client results in a handshake error.

The only difference between the above known-good and known-bad lists is that the bad one has 20 more ciphersuites, all of them DH-DSS-... or DH-RSA-... suites, sprinkled throughout:

DH-DSS-AES256-GCM-SHA384
DH-RSA-AES256-GCM-SHA384
DH-RSA-AES256-SHA256
DH-DSS-AES256-SHA256
DH-RSA-AES256-SHA
DH-DSS-AES256-SHA
DH-RSA-CAMELLIA256-SHA
DH-DSS-CAMELLIA256-SHA
DH-DSS-AES128-GCM-SHA256
DH-RSA-AES128-GCM-SHA256
DH-RSA-AES128-SHA256
DH-DSS-AES128-SHA256
DH-RSA-AES128-SHA
DH-DSS-AES128-SHA
DH-RSA-SEED-SHA
DH-DSS-SEED-SHA
DH-RSA-CAMELLIA128-SHA
DH-DSS-CAMELLIA128-SHA
DH-RSA-DES-CBC3-SHA
DH-DSS-DES-CBC3-SHA

I took each of these cipher suites in turn, supplied it to the server as the first choice in front of DES-CBC3-SHA, and each time the handshake succeeded, picking DES-CBC3-SHA, of course. At this point it seems none of the cipher suites is a poison pill, but there may be just too many for the server. If I take the default cipher list that had the original problem, reorder it so that DES-CBC3-SHA comes first, then the handshake succeeds.

Next, I used this script to exhaustively try moving DES-CBC3-SHA further forward in the cipher suite list. It seems that if that suite is further back than 70th in the list, the server doesn't see it, and it fails the handshake.

It seems we have a DES-only server, parsing the handshake with fixed-length buffers. I'd wager there's an outdated SSL accelerator on the other end.

konklone commented 8 years ago

cc @h-m-f-t for visibility if he has contacts there.

This is excellent detail @divergentdave, thank you. Failing on newer OpenSSL clients is particularly corrosive and bad, this is worth getting them to fix. Hopefully it's a vendor that can fix it for others, too.

h-m-f-t commented 8 years ago

Thanks for the heads up! I will loop ARC in.

divergentdave commented 7 years ago

This regressed recently. It seems the arc.gov server is still TLS 1.0 only, and still requires that one ciphersuite sufficiently far up. I think upgrading requests probably caused this by jostling the ciphersuite list again. I'll take a look at fixing this over the weekend.

Reference: https://github.com/kennethreitz/requests/issues/3774#issuecomment-267871876

divergentdave commented 7 years ago

This guy gets it https://github.com/kennethreitz/requests/issues/3608#issuecomment-250681069

divergentdave commented 7 years ago

The arc.gov server still only supports DES-CBC3-SHA/TLS_RSA_WITH_3DES_EDE_CBC_SHA. I just upgraded my personal computer to Debian Stretch, with OpenSSL 1.1.0f, and support for triple DES has been dropped. (The production server currently runs Ubuntu 16.04 LTS, OpenSSL 1.0.2g, and with DES-CBC3-SHA second to last in the ciphersuite list) If this continues to be a problem, I may try a static build of pyOpenSSL.

@h-m-f-t, could you ping ARC again?

h-m-f-t commented 7 years ago

@divergentdave can do! 3DES cipher's days are numbered in FedGov...