pshtt may not be handling client cert authentication properly

[konklone]

See https://github.com/18F/pulse/issues/758 for the details, but it looks like pshtt may choke when client certs are required and mark it as non-compliant. If this is true, I think we should look into ways to prevent this.

cc @PaulSD

[konklone]

The reference example is sdv.max.gov. Here's GSA's latest pshtt scan for it, showing that port 80 redirects to 443, but the connection is refused on port 443 because we don't attempt to initiate a connection with a client certificate.

So, thinking about this a bit more...

• The requests library does allow a client certificate to be passed with the cert parameter, so in theory we can generate a (fake, invalid) certificate and use it for such requests to measure the connection.
• The error message we get when attempting to requests.get("https://sdv.max.gov") with the usual options provides no information to suggest that a client certificate is required to connect. It's just a flat Connection refused at the libcurl level.
• Similarly, visiting https://sdv.max.gov in Chrome causes an ERR_CONNECTION_REFUSED error. I am not prompted to provide a client certificate (as I have seen in some websites which ask for one).
• Retrying every failed HTTPS connection just to see if it requires a client certificate would be a probably unacceptable performance/bandwidth hit to our scans, as well as extra (unusual-looking) network traffic to federal web services.

Stepping back and looking at it from a policy intent perspective: M-15-13 and BOD 18-01 ask that agencies deprecate insecure traffic and take all available measures to instruct clients to initiate secure connections (and to support those connections once clients make them).

Pshtt evaluates support for secure connections for traditional (server certificate) TLS by making sure port 443 is open and ensuring that the certificate is unexpired and for the intended hostname. The former check is relevant for client authentication, but the latter are not -- so even if we could detect endpoints that expect client certificates, pshtt would not be able to use the same logic path to evaluate the strength of the connection. But when the client provides the certificate, the client is the one that is ensuring authenticity, and so we shouldn't be evaluating the server's obligations there anyway.

That's all to say that pshtt can't really have anything interesting to say about client certificate connections. If they're available at all, then the server's meeting its policy obligations to support them by definition whether or not we scan it, because opening the port is all we could evaluate.

So, we could look into something novel like retrying HTTPS connections another time with a client cert in the specific case of an HTTP redirect that appears to redirect directly to a Connection refused error (as opposed to other certificate/connection error types). However, this is more complexity than we'd ideally need to add to address this case.

@PaulSD: How viable is it to drop the port 80->443 redirect for this hostname? Is it ever expected that that would be used? Dropping port 80 would at least patch the problem, because the domain would be seen as entirely inaccessible over HTTP.

This may also be be compatible with policy intent as a general solution: 80->443 is only allowed for compatibility reasons, so that old links, and the common user behavior of manual link entry, can still be supported even before an HSTS policy takes effect. But these may not be very relevant for the highly specialized (and rare) cases of client-certificate-enabled services, and so it may be acceptable to ask service owners to remove that compatibility redirect.

[konklone]

Another thought: even if we essentially ignore client cert connections from a pshtt / M-15-13 perspective, from a BOD 18-01 perspective we should still be asking that the connection drop support for RC4/3DES/SSLv2/SSLv3.

That would be a domain-scan/sslyze problem, not a pshtt problem, though sslyze may have a better chance of being able to detect the server's need for a client cert than the libcurl-based requests library does.

I'd defer to @h-m-f-t and @jsf9k as to whether they think it's worth looking into the possibility of instrumenting sslyze for this possibility, but I'd also note that if pshtt stays working as it is, then it would be a moot point, because pshtt will still detect the domain as not operative on port 443, which is information the sslyze scanner in domain-scan uses to skip over hostnames without scanning.

[konklone]

Yet another thought: I'm not an expert at TLS connections using client certificates, and thinking about it more, I may be wrong about the server not having obligations to present a valid unexpired certificate during the connection, if the connection is trying to establish mutual authentication. But I also don't know whether we could realistically present a certificate that would get us far enough to evaluate the presented server certificate, if we presented an invalid certificate to the server.

[PaulSD]

Sorry, port 443 on sdv.max.gov was actually down over the weekend, hence the connection refused error ... That's not the error you should normally get.

The way TLS works with client certificates is:

• Client sends "Client Hello" packet to server indicating supported TLS versions, supported Cipher Suites, SNI extension, and some other irrelevant stuff
• Server responds with several messages:
  • "Server Hello" that specifies the selected TLS version, selected Cipher Suite, and some other irrelevant stuff
  • A message containing the server's certificate
  • A client certificate request (which includes a list of CAs whose certificates the server might accept)
  • Some other irrelevant messages
• If the client has a certificate, it responds with the certificate and some additional irrelevant handshake messages. Otherwise, the client will reject the handshake.
• If the server doesn't like the certificate sent by the client, then the server will reject the handshake.

So, for the purposes here, client certificate authentication does not technically prevent checking the server SSL/TLS version, the configured cipher suites, or the server's certificate. Whether or not libcurl or sslyze can return those without completing the handshake is a different question, but it is technically possible to get them. The only thing client cert auth prevents is checking the HSTS header returned in the HTTPS response after the connection is established. However, in this particular case, the base domain is HSTS preloaded, so does it actually matter that you can't check the HSTS header returned after the connection is established?

In this particular case, I could drop the port 80->443 redirect because users never actually type this URL in manually ... However, there are other cases in my environment where client certs are required and the port 80->443 redirect is needed, for example on a domain that unconditionally requires PIV authentication. So that wouldn't be a generic solution.

Another monkey wrench here: This particular URL is actually only used by a specific client-side desktop application that uses its own embedded trust store, so we created an internal CA for it that is only trusted by that client app. So, even after we get past the client cert auth issue, we're still going to run into cert validation issues. However, I think the discussion https://github.com/18F/pulse/issues/760 may cover that issue.

[konklone]

Sorry, port 443 on sdv.max.gov was actually down over the weekend, hence the connection refused error ... That's not the error you should normally get.

...So, for the purposes here, client certificate authentication does not technically prevent checking the server SSL/TLS version, the configured cipher suites, or the server's certificate. Whether or not libcurl or sslyze can return those without completing the handshake is a different question, but it is technically possible to get them.

Ah, okay, this helps -- now it looks like it's a ERR_CONNECTION_CLOSED in Chrome, and a (35) Server aborted the SSL handshake in cURL.

Running requests.get("https://sdv.max.gov", verify=False) gives:

SSLError: HTTPSConnectionPool(host='sdv.max.gov', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

Some quick efforts to generate a fake PEM to use as a client certificate and pass in as the cert parameter for that aren't working well:

Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]

Though I think even if I could figure it out, most of what I said still applies. If killing the port 80 redirect isn't a generic option, this may be tough. Hmm...

The only thing client cert auth prevents is checking the HSTS header returned in the HTTPS response after the connection is established. However, in this particular case, the base domain is HSTS preloaded, so does it actually matter that you can't check the HSTS header returned after the connection is established?

For Pulse's case, and in DHS' scans, this won't be an issue, because preloading a domain automatically grants HSTS compliance for each individual hostname in the domain. (pshtt calculates these separately, though it does return a field about whether the hostname's base domain is preloaded.)

Another monkey wrench here: This particular URL is actually only used by a specific client-side desktop application that uses its own embedded trust store, so we created an internal CA for it that is only trusted by that client app. So, even after we get past the client cert auth issue, we're still going to run into cert validation issues. However, I think the discussion 18F/pulse#760 may cover that issue.

Yeah, you should be fine here as well - it's not required to use a trusted CA, just that it's unexpired and valid for that hostname.

[PaulSD]

Some quick efforts to generate a fake PEM to use as a client certificate and pass in as the cert parameter for that aren't working well:

A fake cert isn't actually going to help ... Assuming the server is properly validating the client cert, it will simply reject your fake cert and terminate the handshake in the same way as it does when you don't provide a cert. A quick test using curl on the command line returns SSL_connect: SSL_ERROR_SYSCALL in connection both for a missing client cert and for a fake client cert that is rejected by the server.

--

Re-reading your earlier comments, this stands out at me as as an indicator of some sort of misunderstanding:

But when the client provides the certificate, the client is the one that is ensuring authenticity, and so we shouldn't be evaluating the server's obligations there anyway.

When client auth is not enabled, the server provides the client a cert, and the client ensures the authenticity of the server cert.

When client auth is enabled, the server provides the client a cert, and the client ensures the authenticity of the server's cert, and also the client provides the server a cert, and the server ensures the authenticity of the client's cert.

So, if pshtt needs to ensure that the server's certificate is unexpired and for the intended hostname, then it needs to do this regardless of whether or not client auth is enabled.

Note that in the TLS exchange described above, the server sends its certificate to the client near the beginning of the handshake. The client should always be validating the server's certificate before reaching the point in handshake processing where client cert auth is relevant. If the server's certificate is invalid, libcurl should fail with an error about that before it fails with any error related to client cert auth.

So, if we can get enough info out of curl to determine that the failure is definitely because of client cert auth (SSL_ERROR_SYSCALL alone seems too generic to be a reliable indicator of that), then it should be safe to assume that curl made it past the server cert validation part...

[PaulSD]

Hmm... To make things more complicated, it looks like different servers behave slightly differently and cause different curl errors. Some servers terminate the connection without terminating the handshake resulting in SSL_ERROR_SYSCALL, other servers send an SSL alert before terminating the connection resulting in error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure...

[konklone]

And when I try hitting it via sslyze (via domain-scan), I get this when pinging sdv.max.gov:443:

_nassl.OpenSSLError: error:140E6118:SSL routines:SSL_CIPHER_PROCESS_RULESTR:invalid command
error:140E6118:SSL routines:SSL_CIPHER_PROCESS_RULESTR:invalid command
error:140E6118:SSL routines:SSL_CIPHER_PROCESS_RULESTR:invalid command

The full command and stack trace is:

$ ./scan sdv.max.gov --scan=sslyze --debug
[375f5b42-c19b-470c-93e1-07a84cdf4e7f] Scan UUID.
[sdv.max.gov][sslyze] Running scan...
    Executing local scan...
    Running scans in serial.
        SSLv2 scan.
        SSLv3 scan.
        TLSv1.0 scan.
    Unknown exception: Traceback (most recent call last):

  File "./scan", line 338, in perform_scan
    data = scan_method(scanner, domain, handles, scan_environment, options, meta)

  File "./scan", line 384, in perform_local_scan
    response = scanner.scan(domain, environment, options)

  File "/home/eric/18f/scan/scanners/sslyze.py", line 106, in scan
    response = run_sslyze(data, environment, options)

  File "/home/eric/18f/scan/scanners/sslyze.py", line 204, in run_sslyze
    sslv2, sslv3, tlsv1, tlsv1_1, tlsv1_2, certs = scan_serial(scanner, server_info, data, options)

  File "/home/eric/18f/scan/scanners/sslyze.py", line 420, in scan_serial
    tlsv1 = scanner.run_scan_command(server_info, Tlsv10ScanCommand())

  File "/home/eric/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sslyze/synchronous_scanner.py", line 52, in run_scan_command
    return plugin.process_task(server_info, scan_command)

  File "/home/eric/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sslyze/plugins/openssl_cipher_suites_plugin.py", line 170, in process_task
    preferred_cipher = self._get_preferred_cipher_suite(server_connectivity_info, ssl_version, accepted_cipher_list)

  File "/home/eric/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sslyze/plugins/openssl_cipher_suites_plugin.py", line 220, in _get_preferred_cipher_suite
    first_cipher = self._get_selected_cipher_suite(server_connectivity_info, ssl_version, first_cipher_string)

  File "/home/eric/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sslyze/plugins/openssl_cipher_suites_plugin.py", line 238, in _get_selected_cipher_suite
    ssl_connection.set_cipher_list(openssl_cipher_string)

  File "/home/eric/.pyenv/versions/3.6.1/lib/python3.6/site-packages/nassl/ssl_client.py", line 313, in set_cipher_list
    self._ssl.set_cipher_list(cipher_list)

_nassl.OpenSSLError: error:140E6118:SSL routines:SSL_CIPHER_PROCESS_RULESTR:invalid command
error:140E6118:SSL routines:SSL_CIPHER_PROCESS_RULESTR:invalid command
error:140E6118:SSL routines:SSL_CIPHER_PROCESS_RULESTR:invalid command

Using cached Public Suffix List...
Results written to CSV.

This looks like a toughie...

[konklone]

@PaulSD can you share another hostname which demonstrates different behavior via cURL from sdv.max.gov

[PaulSD]

Ok, for testing: sdv.max.gov now has a bad server cert, which makes curl barf before hitting the cert auth part. If you tell curl to ignore the server cert error and continue, then the server rejects the missing (or invalid) cert by closing the TCP connection without a TLS alert. sdx.max.gov has a good server cert. The server rejects the missing (or invalid) cert by sending a TLS alert before closing the TCP connection.

[PaulSD]

It looks to me like something is hosed in your sslyze. self._ssl.set_cipher_list(cipher_list) should be called before the SSL handshake starts.

Here's what I get:

$ python3 -m sslyze sdv.max.gov
   sdv.max.gov:443                       => 12.183.70.217   WARNING: Server REQUIRED client authentication, specific plugins will fail.

[konklone]

Hmm, you're right that the sslyze command line interface seems to handle this better than the Python code. I get crashes in Python (and filed an issue about one here: https://github.com/nabla-c0d3/sslyze/issues/294).

It looks like this is going to be tough given pshtt's current code structure, since the SSLyze-based checks only happen after we've already ruled out various TLS handshake errors (we only proceed to the SSLyze section if the connection failed with verify=True but passed with verify=False).

So we could change the logic to run all TLS handshake errors (where it's something more interesting than "port 443 isn't open") through SSLyze as well and try to determine error conditions there. That would require adjusting a decent amount of logic, but might have other benefits in detecting other novel situations (and in just detecting all cases where port 443 is open but not responding in expected ways, which agencies may be interested whether or not certain kinds of configurations are compliant with policies, etc).

Since it'd be a non-trivial change, with the possibility of regressions, I'd love to hear any thoughts from @h-m-f-t or @jsf9k before I take a whack at it.

[jsf9k]

@konklone I think you're right. As long as port 443 is open we should run the host through sslyze.

[PF-Larry]

@konklone @PaulSD

I'm running into this issue as well with workspace.usda.gov.

[jonathanosullivan]

We are seeing this same issue with https://vhs.services.nesdis.noaa.gov which is Citrix NetScaler Access Gateway with Client Certificate Authentication set to mandatory.

[rickminer]

We are seeing this same issue with trident.fisheries.noaa.gov and trident-test.fisheries.noaa.gov. Both have Client Certificate set to mandatory and when scanned return saying that TLS/SSL is not enabled let alone being able to detect an HSTS header.

[PF-Larry]

We have a pseudo workaround for our netscaler, allowing it to be scanned and show complaint without lowering our security posture too much. Be happy to share configurations @jonathanosullivan @rickminer .

[jsf9k]

Client cert issues should be resolved by #179.