Test browsers with user disableable protections disabled

JannisBush commented 2 months ago

In several browsers there are privacy and security features behind a setting that is easy to deactivate by users. For example Brave has Shields and Firefox has ETP.

It is unclear what exactly is controlled by these features and what deactivating it means. Some pages or browsers actively tell users to deactivate these features if the site is broken.

It would be great to also test browsers in a configuration that disables these settings.

arthuredelstein commented 1 month ago

Interesting idea -- relates to the idea of testing opt-in settings as well.

Thorin-Oakenpants commented 1 month ago

I'm not sure if this is feasible or even reasonable - I do not see what it achieves.

It's a bit like sites that want to detect/test if they can check incognito/PB mode - they claim to know but it's not always true, since the factors that determine it can all be toggled individually (at least in gecko), and the test doesn't really serve a purpose (unless you're a website that wants to punish people for not storing state permanently).

For ETP, each individual protection can be customized, and some PB ETP protections can be enabled in all modes. RFP (not front facing) is the same, they can all be toggled on/off via RFPTargets (by not using RFP but FPP and setting preferences). For Brave, in the past, it was trivial for me to detect if the shield was enabled and what level (I haven't bothered for years, bound to still be trivial and that's not the point, Brave is not trying to hide this) - but since strict is now obsolete (or about to be?) and since shields is opt-out by default (right?), and since it's not a guaranteed metric (it can be a per site setting, so needs weighting), I'm not sure what any of these tests would gain

back to @arthuredelstein ... class, discuss!

AlbertoFDR commented 1 month ago

I don't know if this replies your comment but the proposal of this issue if I'm correct is the following one. Each browser has a up/down on/off configuration for privacy/tracking mitigations (shields/ETP). These protections aren't usually documented neither described. It could be that a individual protection is not part of this grouped configuration or that depending on the browser version is included. So, the idea is to test the differences when the protection is up and when it is disabled. This test will allow us to see what could happen if a user disables it for whatever reason (the page is slow, doesn't load...). In other words, it would nice to add, if it makes sense for everyone and it's technically feasible (using profiles?), to check this big-button protections, similar to the comparison of normal vs private mode.

Thorin-Oakenpants commented 4 weeks ago

So, the idea is to test the differences

@AlbertoFDR sure - but for gecko at least, you get those results already in Private Browsing Mode tests. Otherwise (and I am not up to date), you can test your own config

JannisBush commented 4 weeks ago

@Thorin-Oakenpants Gecko ETP has the following configurations: standard (default, tested), strict (more or less the default in private windows I think tested), custom (+ about:config; not tested and I agree testing each possible config here does not make sense), and off (not-tested).

The issue here is about additionally testing the last category on privacytest, i.e., one more config for each browser. I believe it makes sense to test this last category as it is easy for a user to deactivate the protections on a site while it is unclear what this deactivation actually means.

To deactivate the protections a user simply has to click on the ETP or Shields icon in the URL bar and then toggle the on/off button. I suppose a substantial amount of users do this at least occasionally as some sites do not work with the protections on/or actively tell the user to deactivate these features.
It is unclear to a user (or even to researchers) what exactly turning off the toggle means. Sure, one could look at the code to figure out which flags/features exactly are controlled by the toggle, but that is not straightforward.

The button in Brave says:

If this site seems broken, try Shields down. Note: this may reduce Brave's privacy protections.

The documentation then says:

The second layer—our advanced privacy protections—include many features and Chromium customizations built right into the browser. These include reduced network server calls, partitioning, blocked bounce tracking, and more

Some of these are deactivated when Shields is toggled off, whereas other features cannot (easily) be deactivated and are always active in Brave.

The button in Firefox says:

Enhanced Tracking Protection On for this site

And the documentation says:

Note: Total Cookie Protection is enabled by default in Standard mode. This confines every cookie to the website where it was created and prevents cookies from tracking you across sites (to learn more, see Introducing Total Cookie Protection in Standard Mode). Strict Mode also includes Enhanced Cookie Clearing, which allows users to clear third-party cookies more effectively.

In general the documentation reads as it is mainly about cookies, fingerprinting scripts and co., however, other features such as improved privacy in the Referrer-Policy are also toggled with this switch, whereas other features are baked into the browser and are not deactivated.

Adding tests for the off mode would give interested users insights into what exactly toggling this switch means. At the same time it would give browser vendors insights into what features other browser vendors include in the toggle.

There certainly were, are, and will be differences here. Some of them are included as exception in browsers due to web compatibility concerns and I think these decisions are not regularly reevaluated and other browser may come to other conclusions (at a different time).

For example, ETP has two exceptions for unsafe Referrer-Policies (using them for same-site is allowed, and using them for top-level navigations is allowed), Safari only has the first exception, and Shields has no exception (in addition also does not allow origin). (Currently not tested in privacytests as there is only a simple test that checks that document.referrer contains any information)

Also, Brave Shields seems to be applied per origin, whereas ETP seems to be applied per site.

Thorin-Oakenpants commented 3 weeks ago

Gecko ETP has the following configurations: standard ... strict ... custom ... and off ... The issue here is about additionally testing the last category on privacytest

There is no OFF

standard = default
strict = PB mode
custom, quote you: "(+ about:config; not tested and I agree testing each possible config here does not make sense"
off: does not exist

Again, none of this makes any sense

JannisBush commented 2 weeks ago

I guess there seems to be a major misunderstanding here, I try to explain what I mean with this issue report again.

I am talking about deactivating ETP/Shields/similar protections in other browsers on a visited website. You are completely right that in the Firefox settings there is no option available to globally deactivate ETP. In other browsers such options might exist. In Brave there is no simple global on/off button for Shields, however one can deactivate all/many? features that Shields controls individually.

However, what is easily possible is to deactivate ETP/Shields on the currently visited website (persistently; either site or origin-scoped).

Example in Firefox:

Example in Brave:

This issue suggests testing what the outcome of "clicking that button" on the current top-level site has.

Extra Material:

Screenshot of the Brave Shields settings (disabling everything possible):

AlbertoFDR commented 2 weeks ago

@Thorin-Oakenpants can you explain why for you doesn't make sense please.

Thorin-Oakenpants commented 2 weeks ago

Per site exceptions are not universal, and doing so is pretty much the same thing as testing a custom setup - i.e disable for the testing site, either privacytests or a fingerprinting test page (beware all the bullshit entropy figures, just focus on the values returned, and assuming you know what and how a metric is tested, and how to interpret it), and you can already see what changes

By the same argument as OP we should also have every browser at max protection settings. And then some other configuration. And then another. It's insanity :) There is a reason it uses default settings and pb mode - nothing more is needed IMO, given users can self test at some point

The average users is not going to care, or understand. And those who do care it's pretty obvious IMO (read the settings or descriptions), and they can self test at some point

There are far better and more important things that can be added/tested than add all this noise and complexity for users to navigate

privacytests / privacytests.org

Test browsers with user disableable protections disabled #208