Implementing HTTP Strict Transport Security

[nbraud]

Is your feature request related to a problem?

HTTP Strict Transport Security (HSTS) is a security mechanisms enabling HTTP servers to inform clients that all communication with a given host should be performed over a secure TLS channel; per RFC 6797:

A key vulnerability enabled by click-through insecurity is the leaking of any cookies the web resource may be using to manage a user's session. The threat here is that an attacker could obtain the cookies and then interact with the legitimate web resource while impersonating the user.

Jackson and Barth proposed an approach, in ForceHTTPS, to enable web resources to declare that any interactions by UAs with the web resource must be conducted securely and that any issues with establishing a secure transport session are to be treated as fatal and without direct user recourse. The aim is to prevent click-through insecurity and address other potential threats.

This specification embodies and refines the approach proposed in ForceHTTPS. For example, rather than using a cookie to convey policy from a web resource's host to a UA, it defines an HTTP response header field for this purpose. Additionally, a web resource's host may declare its policy to apply to the entire domain name subtree rooted at its host name. This enables HTTP Strict Transport Security (HSTS) to protect so-called "domain cookies", which are applied to all subdomains of a given web resource's host name.

aiohttp would strongly improve security for its users by adopting HSTS, protecting them from a wide variety of attacks (downgrade attacks, SSL stripping, ...)

Describe the solution you'd like

I would like to implement both:

• [ ] in-band HSTS signalling, and;
• [ ] HSTS Preloading.

Both require a datastructure similar to the cookie-jar, to store per-domain policy information, whether it applies to subdomains, etc. This provides a convenient place for users to override behaviour (similarly to what's currently done with `

In-band HSTS Signalling

This is simply what's defined by RFC 6797, but a few things bear hilighting:

• per the specification, ssl=False & similar check-bypasses should be overridden for HSTS destinations:

  2.2.2: The UA terminates any secure transport connection attempts upon any and all secure transport errors or warnings.

• short-lived scripts using aiohttp as a client would greatly benefit from a persistent store for HSTS policies, unlike CookieJar which isn't persisted by default. However, such a store should be application-specific, to avoid creating cross-application leaks, i.e. a user using two aiohttp-based applications, and accidentally revealing through one that they visited some host through the other; this is significant, as the user may have different network configuration for the applications (proxy, VPN or Tor, etc.) or may use them in different contexts (e.g. at work vs. at home). Given there is (to my knowledge) no mechanism for applications to identify themselves (by name, URI, ...) when constructing a Session object, those requirements are incompatible; I would greatly appreciate feedback on possible solutions, and whether there is a better default than either a user-global policy store, or a session-specific one.

HSTS Preloading

The HSTS Preload List is included in major user agents (incl. Chromium & derivative, Firefox, Safari, ...) and provides a set of websites that advertise the HSTS header, with a long duration (cur. 2y or more), and have been voluntarily submitted by their operator. As of now, the preload list contains 162 067, including 33 TLDs, and many major web services.

Preloading has a major advantage: it protects the user's initial request, preventing a MitM adversary from running a successful downgrade attack even if the user never connected to that host before.

Obtaining the authoritative copy of the preload list from Chromium's source repository is straightforward, and so is generating a compact data file sufficient for aiohttp's needs. The difficult part, in my opinion, would be to adjust aiohttp's release process to include the retrieval of the preload list and generation of a static policy file. (I would suggest the file be xz-compressed JSON, as this is straightforward and doesn't require anything beyond the stdlib.)

Describe alternatives you've considered

I considered implementing only HSTS Preloading, and may even do so initially: unlike in-band signalling, there is no complex design trade-off that needs resolving.

Related component

Client

Additional context

HSTS was tangentially mentionned in #3416. Some of the usecases motivating that feature would be addressed by letting users add custom entries to the preload list, when constructing the session.

Moreover, the HSTS Preload list contains additional policy information for CA or certificate pinning; not only would this yield additional security benefits for all users without modifying their applications, but it also provides an obvious API for users to set custom pins (addressing #3679). Note that I believe pinning should be left entirely out-of-scope for this issue: unless something changed ~recently, Python's ssl API has no reasonable affordances for custom cert-validation logic; in principle, I guess one could copy the SSLContext and populate cadata with a restricted set of CAs, but that seems like a a terrible idea. :sweat_smile:

Code of Conduct

I agree to follow the aio-libs Code of Conduct

[Dreamsorcerer]

I think this is an overly complex feature for a mechanism that is probably already on its last legs. If you care about security, just enforce HTTPS on everything, and carefully check anything that claims not to support it.

Firefox already has an option to always use HTTPS and I very rarely find a website where I have to downgrade to HTTP. Given how widespread HTTPS is now, I'd expect this feature to become the default behaviour on most browsers in the next few years, at which point HSTS serves no purpose.

Therefore, to further encourage secure connections, I personally wouldn't be opposed to making HTTP connections disabled by default (in aiohttp v4) and requiring the user to explicitly allow HTTP connections (for all or only a whitelist of domains). But, creating a complex feature to support HSTS seems like a lot of extra maintenance burden for very little benefit.

[nbraud]

I think this is an overly complex feature for a mechanism that is probably already on its last legs. [...] I personally wouldn't be opposed to making HTTP connections disabled by default (in aiohttp v4) and requiring the user to explicitly allow HTTP connections (for all or only a whitelist of domains).

What do you mean? HTTPS enforcement by default is AFAIK not on any major user-agent's security roadmap yet. I assume it isn't on aiohttp's either, given that the relevant issue is inactive since 2018.

If I am mistaken, and there are concrete efforts underway to make aiohttp v4 TLS-only by default, this would indeed change the security/maintainability tradeoff. However, I doubt this is realistic in the current Web ecosystem.

If you care about security, just enforce HTTPS on everything, and carefully check anything that claims not to support it.

First, this proposal is about making all users of aiohttp more secure, by default, not just users who “care about security”.

Moreover, aiohttp is a user agent that is meant to be used programatically; a human might not be available to “carefully check anything that claims not to support [HTTPS]”, so there needs to be a programatic way to perform such a check: that's exactly what HSTS is.

Lastly, pushing the burden of implementing such a check downstream, means that:

• most applications that rely on aiohttp will not do it, and have worse security as a result;
• there will be many disparate implementations, with a higher total maintenance burden, and less ressources (code review, testing, etc.) investing in the security of each one.

Firefox already has an option to always use HTTPS and I very rarely find a website where I have to downgrade to HTTP. Given how widespread HTTPS is now, I'd expect this feature to become the default behaviour on most browsers in the next few years, at which point HSTS serves no purpose.

You seem to be operating under a severe misconception: Firefox's “always use HTTPS” feature (which I also use) does not supplant HSTS, they are complementary; the user agent does not provide an “ignore this issue” click-through on hosts that do not have (working) TLS, and are covered by an HSTS policy.

The absence of click-through is important, because:

• due to alarm fatigue, most human users will click through a security alert if they can;
• even when they do check, humans are notoriously bad at “carefully checking” certificate errors etc. (I could link you the relevant research & user studies, but IIRC the RFC and paper I already linked, directly refer to those)

TL;DR: HSTS serves a purpose, even in an hypothetical world where all user agents require user consent on insecure connections.

[nbraud]

PS: I forgot to mention, but that “overly complex feature” can be implemented in less than ~150 lines of Python, and probably much less if one isn't rolling a custom CRC8 implementation in the process.

[asvetlov]

Sorry, I don't follow how is https://github.com/sethmlarson/hstspreload/blob/main/hstspreload/__init__.py related to HSTS. It even doesn't operate with the mandatory Strict-Transport-Security header.

@nbraud would you make a pull request with desired HSTS API? It could not contain tests at the first stage but should demonstrate what do you want to have at the finish.

[nbraud]

Sorry, I don't follow how is https://github.com/sethmlarson/hstspreload/blob/main/hstspreload/__init__.py related to HSTS. It even doesn't operate with the mandatory Strict-Transport-Security header.

Yes; hstspreload is a library that provides a single function: in_hsts_preload(host: AnyStr) -> bool returns True iff host is in the HSTS Preload list, which is baked into the library at release-time.

Assuming we use it (I don't know aiohttp's stance on dependencies) it would be up to us to add the necessary logic in the client.

@nbraud would you make a pull request with desired HSTS API? It could not contain tests at the first stage but should demonstrate what do you want to have at the finish.

Yes, I was offering to give it a stab, though I might end up having some questions (I'm not so familiar yet with the codebase) ; is Gitter the appropriate place to ask development-related questions?

[asvetlov]

Is Gitter the appropriate place to ask development-related questions?

Sure, you can use gitter or Pull Request comments.

[Dreamsorcerer]

What do you mean? HTTPS enforcement by default is AFAIK not on any major user-agent's security roadmap yet. I assume it isn't on aiohttp's either, given that the relevant issue is inactive since 2018.

Given the push for HTTPS everywhere, including displaying "Not Secure" in the address bar, disabling various features/APIs on HTTP pages, and the introduction of Lets Encrypt; I'd say it's pretty obvious that we are rapidly moving towards an HTTPS only world, and therefore I'd rather get ahead of the curve and add this simpler feature than try to deal with the more complex HSTS feature (bear in mind that aiohttp v4 is probably not going to be released soon).

PS: I forgot to mention, but that “overly complex feature” can be implemented in less than ~150 lines of Python

The complexity is more in the maintenance and user interaction. You've already highlighted several complications:

An HSTS feature would provide little benefit without some way of persisting the state. But, aiohttp is a library, not an application, so it does not handle storing user data. Therefore, we'd need some mechanism for an application to restore this persistent state, which then makes it an opt-in feature, thus no longer improving security (significantly) for all users.

If we go with HSTS preloading, then we have to start shipping a large dataset along with the library, which probably isn't desirable. We also need to figure out a way to update the dataset regularly.

Given these complications, a blanket HTTPS only by default would be much simpler to implement and maintain, while providing better security as it does not require a website to be in the preloaded list or to be previously visited by the user.

I'm not saying HSTS is a bad idea, but given the complexities and drawbacks of implementing it, I'm just not sure if it is worth it. I'd suggest coming up with solutions to the previously mentioned difficulties before writing any code, as it would be a shame to spend that time writing a feature if we then can't merge it.

[Dreamsorcerer]

Given the lack of activity, I think we can say this won't get implemented. Enforcing HTTPS by the developer is also pretty easy, just check the scheme of any URLs passed in.