sslyze results missing for some domains

PaulSD commented 6 years ago

Example: pivcheck1.max.gov and pivcheck2.max.gov are identical (those are two names for the same server/app), but pulse currently shows sslyze data for pivcheck1 but not pivcheck2.

Any idea what is going on there?

konklone commented 6 years ago

Here's what we saved from our SSLyze scan results for those two domains last night:

It looks like, for whatever reason, SSLyze was able to connect to pivcheck1 but not pivcheck2. When I try, just now, to scan pivcheck2 with SSLyze, it works fine.

I wonder if perhaps your firewalls dropped our connection to pivcheck2 because it was happening shortly after or concurrently with our connections to pivcheck1 from the same source. We run these in a pretty high-density manner (900 concurrent Lambda executions!) without rate limiting, relying on the general federated/disparate nature of federal infrastructure to avoid DDoSing servers. And the SSLyze negotiations for each scan are fairly intense, because it tries out each protocol version in turn.

We've seen the same behavior exhibited from some other federal services, and I suspect it's the same dynamic.

For the record - the results I got from sslyze'ing pivcheck2 just now:

{
  "certs": {
    "any_sha1_constructed": false,
    "any_sha1_served": false,
    "constructed_issuer": "Entrust Root Certification Authority - G2",
    "key_length": 2048,
    "key_type": "RSA",
    "leaf_signature": "sha256",
    "not_after": "2020-03-20T21:42:36",
    "not_before": "2017-11-13T21:12:38",
    "served_issuer": "Entrust Root Certification Authority"
  },
  "config": {
    "all_dhe": false,
    "all_rc4": false,
    "any_3des": false,
    "any_dhe": true,
    "any_rc4": false,
    "weakest_dh": 256
  },
  "errors": "",
  "hostname": "pivcheck2.max.gov",
  "protocols": {
    "sslv2": false,
    "sslv3": false,
    "tlsv1.0": true,
    "tlsv1.1": false,
    "tlsv1.2": true
  }
}

konklone commented 6 years ago

@PaulSD In last night's run, both pivcheck1 and pivcheck2 returned sslyze results. I didn't make any changes on this end, did you?

In an near term update, SSLyze is dropping the threads for cipher evaluation from 15 to 10: https://github.com/nabla-c0d3/sslyze/commit/6fa29898a19f469586cf4c6a018f0f0651eb04c1 Which could help reduce this sort of churn (at the cost of slightly slower scans), if it is about just getting dropped for having too many open connections in too short a period of time.

PaulSD commented 6 years ago

I haven't changed anything...

konklone commented 6 years ago

I'm going to start capturing some more information going forward here: https://github.com/18F/pulse/pull/762

Since it's intermittent, I'll leave this open for a bit, and treat this as a reference issue for seeing if I can get more data about intermittent sslyze failures in general, whether they affect pivcheck2.max.gov or not.

PaulSD commented 6 years ago

Interesting to note: On Friday, pivcheck1 and pivcheck3 had results but pivcheck2 did not. Today, pivcheck1 and pivcheck2 have results, but pivcheck3 does not. All three of those are the same.

I see no evidence on my side that this traffic tripped any rate limits.

PaulSD commented 6 years ago

Has #762 been deployed? Can you point me at the output from last night?

konklone commented 6 years ago

Yes it is -- here are the latest sslyze scan results:

https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/live/subdomains/scan/results/sslyze.csv

PaulSD commented 6 years ago

"Connection timeout while talking to Lambda. Scan returned nothing."

There are relatively few of these in the first 10000 or so scans, then tons of them for the next 5000 scans, then a modest number in the last 8000 scans. Only one of my domains failed last night, and it failed with this error.

Perhaps you are hitting rate limits between the system that is coordinating the scans and AWS?

18F / pulse

sslyze results missing for some domains #759