AIA chasing for missing intermediate certificates on TLS connections

[dstufft]

BPO	18617
Nosy	@april, @pitrou, @tiran, @alex, @dstufft, @yan12125, @ofek

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/tiran' closed_at = None created_at = labels = ['expert-SSL', '3.8', 'type-feature', 'library'] title = 'AIA chasing for missing intermediate certificates on TLS connections' updated_at = user = 'https://github.com/dstufft' ``` bugs.python.org fields: ```python activity = actor = 'Ofekmeister' assignee = 'christian.heimes' closed = False closed_date = None closer = None components = ['Library (Lib)', 'SSL'] creation = creator = 'dstufft' dependencies = [] files = [] hgrepos = [] issue_num = 18617 keywords = [] message_count = 9.0 messages = ['194088', '194090', '203174', '203175', '216976', '293894', '293896', '293898', '386160'] nosy_count = 10.0 nosy_names = ['janssen', 'april', 'pitrou', 'christian.heimes', 'alex', 'docs@python', 'dstufft', 'samv', 'yan12125', 'Ofekmeister'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue18617' versions = ['Python 3.8'] ```

[dstufft]

Currently the Python SSL module requires the full chain, including all intermediaries, to be served in order to validate a TLS connection. This isn't *wrong* however a number of folks have had issues trying to setup a custom PyPI index in pip due to missing them. The believed the problem with validation to be in pip/Python because it works in their browser.

Essentially browsers have the ability to download missing intermediate certs using an url found inside the AIA data. This is slow and isn't generally recommended that you rely on it for any particular site. However it does mean that TLS connections work in more situations.

At the least this should probably be documented, and possibly the Python library be enhanced to be more browser like in this regard.

[tiran]

Thx!

Let's update the docs for 2.7 to 3.4 first.

For AIA intermediate cert retrieval we have to use OpenSSL's verify hook. It's a pain to implement all necessary bits and hard to use, though. :(

[tiran]

Donald, could you please provide a doc update that explains the problem?

[pitrou]

I'm afraid downloading certs on the fly would open a whole new can of worms, so I'd rather have it documented indeed :)

[c42fb415-8b45-4800-8221-c068c723cac0]

Perhaps the simplest thing here is to add a standard verify callback that catches verification errors, and returns the parsed server certificate as an attribute of the raised exception object. From python, the exception can be caught and then the certificate data info used to fetch the intermediate certificate, and pass it into SSLContext.load_verify_locations().

This would force an extra client connection, but be less insane than trying to fetch and return the certificate from inside the SSL_CTX_set_verify() callback IMHO.

Does that sound workable? Any hints for a would-be drive-by patcher?

[caf15ad9-e015-4ae9-bbad-9922fd74e99b]

Browsers universally support AIA chasing/fetching, as do a number of underlying OS libraries, such as Secure Transport (macOS) and schannel (Windows).

As a result, it is becoming increasingly common for server operators to fail to include the entire certificate chain in their handshake. By some measures, anywhere from 10 to over 30% of all TLS errors(1) are caused by missing intermediate certificates.

There are a handful of ways that scanners and other tools have worked around this issue. Some, like SSL Labs, do their own AIA chasing. Others make subprocess to other tools linked to OS libraries which invisibly perform AIA chasing, such as curl on macOS. Others, like Firefox, maintain a cache of intermediate certificates.

In any case, all of these solutions are concession to the reality that TLS is really hard and that it is largely impossible to solve this particular misconfiguration issue.

I myself write a popular tool(2) that does scanning of sites for missing security configurations. Although it's not explicitly focused on TLS, some of the underlying tests rely on a verified TLS connection. For example, the HTTP Strict Transport Security (HSTS) header can only be set on valid TLS connections. I use requests (which uses httplib underneath), but I'm unable to tell if a given certificate error is because of either:

a) the certificate validation actually failed (for example, due to being expired), or b) there was a missing intermediate certificate and that things would have worked if AIA chasing was available

As a result, I have to tell people that their HSTS header can't be set. This is often incorrect, because browsers that do AIA chasing consider the connection perfectly secure for the purposes of HSTS.

It would be really nice if Python's httplib could be updated to do transparent AIA chasing, with the optional ability to turn it off. In an ideal world, the underlying TLS libraries (such as OpenSSL) would do this for us, but unfortunately as things stand now OpenSSL leaves it up to the applications themselves to perform chasing. According to Mozilla research (3), if implemented, it would reduce TLS errors by approximately 6% were AIA chasing always performed and successful.

Thanks!

1. https://docs.google.com/document/d/1ryqFMSHHRDERg1jm3LeVt7VMfxtXXrI8p49gmtniNP0/edit?pli=1
2. https://observatory.mozilla.org/
3. https://bugzilla.mozilla.org/show_bug.cgi?id=399324#c80

[caf15ad9-e015-4ae9-bbad-9922fd74e99b]

Err, sorry, I may have been a bit unclear. AIA chasing is not *universal* amongst browsers. IE, Edge, Chrome, and Safari perform AIA chasing. Firefox maintains a cache of intermediate certificate authorities. I noted that a bit further underneath, but my apologies if it was confusing.

That said, every browser has a some chosen some underlying method to ameliorate this common TLS error. It would be amazing if Python could similarly resolve this issue for developers.

[alex]

Just discussed with Christian, and we're both in favor of adding AIA chasing support to the stdlib ssl.

[73219d1f-a633-4d07-b832-359bf52673c8]

Any update on this?

[kousu]

curl on macOS

Did Apple patch curl to add this? As of today, upstream curl does not do AIA chasing:

https://github.com/curl/curl/blob/bff6277679f7e21bbc421c3fd40f17848950813e/docs/TODO#L825-L834

They closed the request four years ago and haven't taken a look at it since.

[fazalmajid]

@kousu I believe Apple's /usr/bin/curl is compiled using Apple's Secure Transport, which apparently does AIA chasing. You can verify this using otool -L /usr/bin/curl, which has no references to OpenSSL.

[lightninghydra24]

pwetty pwease?

     🥺 
   👉🔒👈 

[milahu]

has workaround: danilobellini/aia

[sevaa]

Encountered a related issue that would also be resolved by the logic of following the AIA during the handshake.

In my case, the authority has reissued their intermediate cert with the same key, the HTTPS server was serving up the trust chain with an expired intermediate cert, Chrome had no problem with that (because the updated one was available via the AIA/CA Issuer URL), but urllib3 would fail, citing an expired authority cert. Clients based on Windows' Schannel (e. g. the Windows build of curl, or .NET's HTTPS client) would work too.