search

chromium / hstspreload.org

:lock: Chromium's HSTS preload list submission website.
https://hstspreload.org
BSD 3-Clause "New" or "Revised" License
782 stars 92 forks source link

Use semantic preload "policies" #111

Closed lgarron closed 7 years ago

lgarron commented 7 years ago

The bulk of HSTS preloaded entries are submitted through hstspreload.org, and are subject to the submission requirements there. However, the submission requirements for bulk entries have changed over time, and we have domains that are submitted under a variety of policies. I would summarize our current set of policies as:

If we annotate each entry with a policy in the Chromium source, programs consuming the preload list can easily filter out bulk entries, or also filter out more fine-grained sets of domains.

TODO: document exact requirements for a domain to be added or removed under each policy.

Use Cases

Removals

We currently use use a hacky script to calculate bulk entries, so that hstspreload.org can accept automated removal requests for bulk domains but not other domains. Since I don’t trust this script a lot, it only runs during deployment, and can get stale. We would like something more automatic and robust: https://github.com/chromium/hstspreload.org/issues/108

Status Info

If you enter a bulk domain on the front page of hstspreload.org, the site will tell you whether the site still meets the continued requirements. However, we are currently calculating this based on the bulk-1-year policy, even if the domain was submitted when the required max-age was only 18 weeks. To keep this feature, we need to be able to tell them apart.

Auto-Roller

If we introduce an auto-roller that updates the Chromium source code based on pending additions and removals domains at hstspreload.org, it should only touch bulk entries. This makes it safer to run without supervision, even if there are bugs or attacks on hstspreload.org

Statistics

When calculating statistics, it it useful to be able to look at trends among just bulk domains, or just domains with a particular bulk policy.

Firefox roller: Google entries

The Firefox script currently preloads all Google domains. However, it does this by looking at which domains are pinned with Google’s HPKP pins, which technically has a different meaning and may go away.

Firefox roller: eTLD and eTLD-requested entries

We would like Firefox to be able to pick up etld and etld-requested-gov sites without scanning them. This is easy if we give them a simple field to look at.

HTTPS EVerywhere

See https://github.com/EFForg/https-everywhere/issues/13235

lgarron commented 7 years ago

Landed in https://chromium.googlesource.com/chromium/src/+/4bf96c30121ed23f8025e0a2ee6c68831fd0786f

Mozilla bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1418112

Bisaloo commented 6 years ago

Could you also expand on the priority rules used to pick the policy for one domain?

For example, youtube.com could be in bulk-1-year but it is listed as google. So I am guessing this means google has a higher priority than bulk-1-year.

What about the rest?

lgarron commented 6 years ago

So I am guessing this means google has a higher priority than bulk-1-year.

What about the rest?

It's not really about priority. Each entry was added under a particular policy, and the annotations simply document that.

Google domains are a bit special, since there are a whole bunch of them and many of them are pinned. We know out-of-band that the entries are valid, so I didn't mark them as bulk (which might accidentally allow them to be removed, and also gives a semantic way for Mozilla to avoid filtering them out). I expect that would continue in the future.

Do you have particular questions about the rest?

Hainish commented 6 years ago

@lgarron

TODO: document exact requirements for a domain to be added or removed under each policy.

Have you had time to address this TODO item? This will help guide our own removal process in HTTPS Everywhere's hsts-prune utility.

Hainish commented 6 years ago

cc @mozkeeler I'm interested in the Firefox removal process as well.

lgarron commented 6 years ago

Have you had time to address this TODO item? This will help guide our own removal process in HTTPS Everywhere's hsts-prune utility.

For the time being, nothing is being removed yet. 🤷

mozkeeler commented 6 years ago

At the moment, I believe Firefox will only remove a site from the list if Firefox can successfully connect to it over https but it doesn't send the Strict-Transport-Security header or it specifies a max-age that isn't large enough.