search

arkenfox / user.js

Firefox privacy, security and anti-tracking: a comprehensive user.js template for configuration and hardening
MIT License
10.27k stars 519 forks source link

tracking: HTTPS-Only mode #1047

Closed Thorin-Oakenpants closed 3 years ago

Thorin-Oakenpants commented 4 years ago

edit: no longer tracking/updating since May 2021

if you see HOM in this issue, it this stands for HTTPS-Only mode

I've gone through the bugzillas and picked the eyes out. Note that we block mixed active and passive by default, and there is also the upcoming insecure downloads pref: these probably all have a bearing on how this works

update: FF83+ the UI is exposed

🔻 meta

🔻 landed

🔻 of interest

🔻 misc / open

🔻 visuals

earthlng commented 3 years ago

this can leak things

correct me if I'm wrong but AFAIK this leak is only really a leak if you use DoH+ESNI. And even then, if a site is only available over HTTP and you end up adding an HOM exception or click the "go there anyway" button then this can not be considered a leak either.

As I understand it, the background request is also only triggered if a webserver doesn't respond to the HTTPS request within 3 seconds. I only mention this because in another issue @rusty-snake said that this "always leaks" and that's not true AFAIK. Most servers will probably respond in under 3 seconds.

we should add this as false to the user.js

we can add it inactive, for people using DoH+ESNI.

rusty-snake commented 3 years ago

As I understand it, the background request is also only triggered if a webserver doesn't respond to the HTTPS request within 3 seconds. I only mention this because in another issue @rusty-snake said that this "always leaks" and that's not true AFAIK. Most servers will probably respond in under 3 seconds.

It was here.

My understanding up to now was that to each server a http and a https connection attempt is made when opening a new site. But thanks to your push I checked this again, and yes it looks like it's only done after 3secs of no response.

https://phabricator.services.mozilla.com/D85300 https://searchfox.org/mozilla-central/source/dom/security/nsHTTPSOnlyUtils.cpp#25

correct me if I'm wrong but AFAIK this leak is only really a leak if you use DoH+ESNI.

The IP of the site is always leaked w/o a VPN. And the domain is leaked if you don't use DoH/DoT/… and w/o eSNI (which needs to be supported by your firefox and the server) it's leaked too.

As long as background-requets are enabled, a attacker (e.g. your ISP) could delay all your requests and then catch these http requsts. However, you would note if every site takes much longer.

As HOM now is on about:preferences I wouldn't consider it experimental anymore (as you said). And it works with FPI (but see below). We should add it as true to the user.js. It will upgrade a lot of site which does not have https by default and if you visit one of the ten-site w/o any https, it's just one click.

FWIW: If you set privacy.firstparty.isolate.use_site=true (and FPI), exceptions don't work. (I guess this is expected, since use_site makes FPI (and therefore site-permissions/exceptions) reflecting the scheme (i.e. http://example.com != https://example.com).

earthlng commented 3 years ago

Thanks for mentioning the problem with use_site, I wasn't aware of that and I doubt anyone at mozilla is either. At least I haven't seen a ticket for that yet. AFAICT use_site is still in a very early experimental state and breakage is to be expected. But yeah, that's definitely something worth mentioning in our notes/warnings for the HOM pref.

rusty-snake commented 3 years ago 
/* 1244: enable HTTPS-Only mode [FF76+]
 * When "https_only_mode" (all windows) is true, "https_only_mode_pbm" (private windows only) is ignored
 * [NOTE] If the last pref is true, firefox sends a http requests after 3 seconds of no response to this _domain_
 * to see if the site supports https. This avoids waiting for timeouts.
 * [SETTING] to add site exceptions: Page Info>Permissions>Use insecure HTTP (FF80+)
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [1] https://bugzilla.mozilla.org/1613063 [META]
 * [2] https://bugzilla.mozilla.org/1647829 ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]
   // user_pref("dom.security.https_only_mode.upgrade_local", true); // [FF77+]
user_pref("dom.security.https_only_mode_send_http_background_request", false); // [FF82+]

Explain how it's not a leak. Do you mean because of DNS queries. i.e the background request only leaks the eTLD?

Since FF82 it only leaks the domain , but not the site. If http://example.com/ has no https, then http://example.com/spam/egg.html too. And the domain is leaked via DNS if no DoH/DoT is used. And w/o ESNI/ECH it's leaked too. So it's not a real leak if you leak anyway.

annevk commented 3 years ago

Could you file a bug on the potential bad interaction between dFPI and HTTPS-only? It's a bit hard to make out what the problem is without context.

earthlng commented 3 years ago

@Thorin-Oakenpants

FYI, these are 2 different prefs!

Did they change the timings?

yes they increased it to 3 seconds

rusty-snake commented 3 years ago

STR (used Nightly 85.0a1 (2020-11-18)):

Test sites:

  1. http://speedofanimals.com/ - no https at all (only on www.speedofanimals.com) = timeout
  2. http://neverssl.com/ - bad cert
  3. http://http.badssl.com/ - working https + downgrade using 301 Moved Permanently = infinity loop
  4. http://randomwalker.info/ no https at all = timeout
  5. http://neverssl.org/ - bad cert

NOTE: You need to open the http url. If you open a https url, HOM is not activated and therefore you see no warning regarding HOM and you can not add a exception.

Nothing changed (in Nightly this means network.cookie.cookieBehavior=5 and privacy.dynamic_firstparty.use_site=true):

Enable HOM (and sanitize cookies, cache, ...):

Enable FPI (and sanitize):

EDIT: Summary: dFPI is not affected. Only HOM + FPI + FPI.use_site.

rusty-snake commented 3 years ago 
/* 1244: enable HTTPS-Only mode [FF76+]
 * When "https_only_mode" (all windows) is true, "https_only_mode_pbm" (private windows only) is ignored
 * [SETTING] to add site exceptions: Page Info>Permissions>Use insecure HTTP (FF80+)
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [1] https://bugzilla.mozilla.org/1613063 [META]
 * [2] https://bugzilla.mozilla.org/1647829 ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]
user_pref("dom.security.https_only_mode_send_http_background_request", false); // [FF82+]
/* 1245: Upgrade requests to local resource too [FF77+] ***/
   // user_pref("dom.security.https_only_mode.upgrade_local", true);
/* 1246: Disable http background requests [FF82+]
 * If a site is upgraded by HOM and the server didn't response with in 3 seconds, firefox sends a
 * http requests to the domain in order to see whether the server supports https or no.
 * This is done to avoid waiting for a timeout which takes 90 seconds.
 * [1] https://bugzilla.mozilla.org/1642387
 * [2] https://bugzilla.mozilla.org/1660945 ***/
user_pref("dom.security.https_only_mode_send_http_background_request", false);
crssi commented 3 years ago

I like http://neverssl.org/, because its random, avoiding cache "problems".

rusty-snake commented 3 years ago

@Thorin-Oakenpants #959

Edit: thoughts on testsites:

xtsm commented 3 years ago

So, is there no proper way to add exceptions/whitelists yet? Because there is no "Use insecure HTTP" line in Permissions tab (FF 86.0.1)

xtsm commented 3 years ago

It doesn't work with some sites that act like they have a HTTPS version but this version is actually a stub asking you to use HTTP instead (e.g. this one). I'm not sure what's the reasoning behind such setups. Guess I'll wait for the UI, thanks.

xtsm commented 3 years ago

Yep, manual SQL insert fixes the issue for websdr.space and several other sites, thanks.

Thorin-Oakenpants commented 3 years ago

I'm done tracking this [in OP]: you're going to get weird edge cases until the interwebs get their shit together and HTTP is outlawed

Thorin-Oakenpants commented 3 years ago

@xtsm ... https://bugzilla.mozilla.org/show_bug.cgi?id=1662710 landed

eleius commented 3 years ago

I was wondering if HOM also upgrades insecure sub-requests, like HTTPS Everywhere does? (I'm looking for an alternative that is not list-basted... btw I thought I'd comment here to avoid spamming the issues section)

rusty-snake commented 3 years ago

I'm not sure if it upgrades sub-requests. Anyway in the worst case you end up with not loading sub-request due to mixed-content blocking.

https://github.com/arkenfox/user.js/blob/00fa8f1b50dbc717ef9cc702679f3829d91fee05/user.js#L692-L697

crssi commented 3 years ago

When I have tested last time looong time ago, when HOM is enabled also mixed passive content is upgraded to HTTPS. I haven't tested for mixed active content, since its blocked anyway (also as default).

But you can easily test your scenario at https://mixed.badssl.com/.

Cheers

crssi commented 3 years ago

Even if you set:

/* 1241  */ user_pref("security.mixed_content.block_display_content", false);

the 1241 setting is ignored when HOM enabled.

image

As you can see in page source the IMG is requested over HTTP, but network console clearly shows that IMG was downloaded over HTTPS

eleius commented 3 years ago

As you can see in page source the IMG is requested over HTTP, but network console clearly shows that IMG was downloaded over HTTPS

So it does upgrade mixed content sub-requests, and blocks them if downloading via https is not available?

Thorin-Oakenpants commented 3 years ago

HTTPS-First is the one that silently fails and allows insecure HTTPS-Only mode (HoM) attempts to upgrade everything, and never allows insecure without intervention: it complains on top-level load. If top level is HTTPS it will silently fail sub-resources after upgrading - it literally does what it says HTTPS ONLY

So it does upgrade mixed content sub-requests, and blocks them if downloading via https is not available?

HoM upgrades everything: this comment says (emphasis mine), and chris makes a point of difference with HTTPS-First not upgrading subresources, being top-level loads only

If HTTPS-Only is enabled, all connections will be upgraded to https

There was also another comment Chris made some time after that, but I can't find it, in which I'm almost certain he did state it does upgrade all subresources (and silently fail)

So that's the intent, AFAIK. If you follow the meta ticket, there are still a few holes (not web content itself that I know of: it's edge cases likeTCP, HSTS) and teething issues here and there, and tickets for finishing building tests - I'm not really following it TBH.

HoM should be pretty robust. IDK if HTTPS-E in EASE Mode is it's equal - given that it's limited by WebExt

eleius commented 3 years ago

HTTPS-Only mode (HoM) attempts to upgrade everything, and never allows insecure without intervention: it complains on top-level load. If top level is HTTPS it will silently fail sub-resources after upgrading - it literally does what it says HTTPS ONLY

Excellent, thanks!

Thorin-Oakenpants commented 3 years ago

re-opening to remind me: we should add that info to the item in the user.js

eleius commented 3 years ago

Btw I'm unable to set exceptions, either from the padlock icon (on/off) or from the "manage exceptions" section in about:preferences#privacy, probably because of FPI as reported by @rusty-snake above. You might want to add this info too.

Thorin-Oakenpants commented 3 years ago

worksforme, also where I don't see where rusty snake-eyes g.i. joe said something about FPI, but yes, you need to make sure you exception has the right FPI syntax

can you give me the site that doesn't work for you?

edit: you add the exception via the padlock while you are on the interstitial BEFORE you click continue to HTTP Site ignore that

example: here's my only exception

Thorin-Oakenpants commented 3 years ago

hmmm, I just tried in Nightly and got this garbage (which doesn't work) - it's using the interstitial as the 1stparty by the looks of it garbage

What version of FF are you on? 91? I wonder when it broke for FPI

eleius commented 3 years ago

also where I don't see where rusty snake-eyes g.i. joe said something about FPI

Last 3 lines in this comment

can you give me the site that doesn't work for you?

Any http website, e.g. libgen.lc - I add the exception (usually setting to "off" via padlock, I also tried via manage exceptions), shutdown Firefox, and the next time there is no exception listed in "manage exceptions".

Since your example (impawards) also met the same fate, I think one of my addons is deleting them for some reason. I'll have to do some tests...

rusty-snake commented 3 years ago

@eleius Either don't set privacy.firstparty.isolate.use_site=true or use dFPI.

Did you set privacy.clearOnShutdown.siteSettings=true?

rusty-snake commented 3 years ago

http://http.badssl.com/ -> "continue to HTTP Site" -> Padlock "Not Secure" -> HTTPS-Only mode -> Off

WFM with FPI=true FPI.use_site=false

rusty-snake commented 3 years ago

If you set a HOM exception from the Padlock, it uses the current site. If you are on about:httpsonlyerror it will use http://example.com^firstPartyDomain=about:httpsonlyerror (fake syntax) and if you have clicked on "continue to HTTP Site", it will use http://example.com^firstPartyDomain=example.com.

eleius commented 3 years ago

Did you set privacy.clearOnShutdown.siteSettings=true?

Ah yes. So https exceptions are counted as "site settings".

rusty-snake commented 3 years ago

Permissions are site settings.

rusty-snake commented 3 years ago

Here's the full list:

Ci.nsIClearDataService.CLEAR_PERMISSIONS |
  Ci.nsIClearDataService.CLEAR_CONTENT_PREFERENCES |
  Ci.nsIClearDataService.CLEAR_DOM_PUSH_NOTIFICATIONS |
  Ci.nsIClearDataService.CLEAR_SECURITY_SETTINGS |
  Ci.nsIClearDataService.CLEAR_CERT_EXCEPTIONS

https://searchfox.org/mozilla-central/source/browser/modules/Sanitizer.jsm#513-517

Thorin-Oakenpants commented 3 years ago

IDK what's going on. In FF85 or 86 or whatever or 89 or 90 it worked, for the impawards site, single upgrades worked with FPI, adding exceptions worked with FPI (e.g. edit in sqlite if needed). When the UI for exceptions came along, it still all worked. It still works in my FF91 main Firefox.

But now it all seems broken with FPI in nightly. Seriously, I am so tired of all this shit (not you guys, but the constantness of all this stuff), that honestly, I think I might just fuck off for a year .. I've almost had it

@rusty-snake

and if you have clicked on "continue to HTTP Site", it will use http://example.com^firstPartyDomain=example.com

FPI is enabled. Clicking continue to HTTP on the interstitial is also broken and just loops fpi

If FPI is disabled, everything seems to work.

eleius commented 3 years ago 
Ci.nsIClearDataService.CLEAR_CONTENT_PREFERENCES

Ok I'm setting it to false, I just hope "content preferences" (?) cannot be abused for tracking.

Thorin-Oakenpants commented 3 years ago

OK, so it was you clearing site preferences on close, then? right? thank fuck for that.

As for nightly, fuck it. I just tested in my main FF and everything works fine with FPI, so until breakage happens in stable, I do not care. Taking a year off, see you in 2022

eleius commented 3 years ago

Oh man, sorry about that...

Thorin-Oakenpants commented 3 years ago

Oh man, sorry about that...

not your fault. I assumed you wouldn't be deleting site exceptions (we don;t in the user.js) because that's rather extreme. And I just jumped to the most logical conclusion that it was one of those weird edge case sites: hence why I asked what it was

I just hope "content preferences" (?) cannot be abused for tracking.

"content preferences" ? If you mean site exceptions, then no. Exceptions are not global (they are literally exceptions) so can't be used to help track you, and exceptions are only things like allow/prompt/block things - do a control-I on a page and view permissions

eleius commented 3 years ago

I assumed you wouldn't be deleting site exceptions (we don;t in the user.js) because that's rather extreme.

But I shouldn't have posted here using a customized js, I totally forgot I had switched that one to true.

Thorin-Oakenpants commented 3 years ago

pants

edit: you add the exception via the padlock while you are on the interstitial BEFORE you click continue to HTTP Site ignore that

rusty

f you set a HOM exception from the Padlock, it uses the current site. If you are on about:httpsonlyerror it will use http://example.com^firstPartyDomain=about:httpsonlyerror (fake syntax) and if you have clicked on "continue to HTTP Site", it will use http://example.com^firstPartyDomain=example.com.

sorry rusty, I simply don't use the interface or add exceptions (except the one I have had for ages) - so my method was a little RUSTY (sorry). We better add that: edit bit pointless listing off temporarily then isn't it, in order to use it you already set that (talk about ass about face)

 * [SETTING] to add site exceptions: Padlock>HTTPS-Only mode>On <-- when on the site u MOFOs
 * [SETTING] Privacy & Security>HTTPS-Only Mode
Thorin-Oakenpants commented 3 years ago

OK, how is this

@eleius @rusty-snake 

/* 1244: enable HTTPS-Only mode in all windows [FF76+]
 * If the top-level is HTTPS, then insecure subresources are also upgraded, failing silently
 * [SETTING] to add site exceptions: Padlock>HTTPS-Only mode>On <- when on the site
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [TEST] http://example.com [upgrade]  <--- RUSTY FIND ME SOMETHING ELSE
 * [TEST] http://neverssl.org/ [no upgrade]
 * [1] https://bugzilla.mozilla.org/1613063 [META] ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]
rusty-snake commented 3 years ago 
/* 1244: enable HTTPS-Only mode in all windows [FF76+]
 * If the top-level is HTTPS, then insecure subresources are also upgraded, failing silently
 * [SETTING] to add site exceptions: Padlock>HTTPS-Only mode>On (After "Continue to HTTP Site")
 * [SETTING] to add site exceptions: click "Continue to HTTP Site", Padlock>HTTPS-Only mode>On
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [TEST] http://example.com [upgrade]  <--- RUSTY FIND ME SOMETHING ELSE; Actually every site with ~correct configured~ _working_ https, _idealy without 301 upgrade_. However if it is in https://hg.mozilla.org/mozilla-central/raw-file/tip/security/manager/ssl/nsSTSPreloadList.inc, it will be upgrade regardless of HOM. Will look for a good one.
 * [TEST] http://neverssl.com/ [no upgrade]  <--- .org is dead
 * [1] https://bugzilla.mozilla.org/1613063 [META] ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]
Thorin-Oakenpants commented 3 years ago

shit that was meant to be the neverssl one

Thorin-Oakenpants commented 3 years ago

wait a second ... http://neverssl.com/ is working - https://github.com/arkenfox/user.js/issues/1235#issuecomment-901253176

so I guess we don't need to worry about that now