search

biancadanforth / tracking-protection-shield-study

A Shield study to determine the optimal messaging, if any, for Tracking Protection in Firefox.
0 stars 3 forks source link

Shield Study contradicts Tracking Protection UI in about:preferences #194

Open dholbert opened 6 years ago

dholbert commented 6 years ago

(Not sure if this is just a limitation of Shield Studies vs. native browser UI, but I figured I'd submit because it seems confusing...)

STR:

  1. Receive the Tracking Protection shield study (I just randomly did, in a new Firefox profile)
  2. Visit e.g. Facebook.com, notice the Tracking Protection URL icon, and choose "Keep Protecting Me"
  3. Now visit about:preferences in a new tab and search for "Tracking"

ACTUAL RESULTS: The about:preferences says "Use tracking protection: [...] Never". This contradicts what the icon in the URL bar at Facebook told me.

EXPECTED RESULTS: Non-contradictory messaging from Firefox about whether or not I'm being protected from trackers.

dholbert commented 6 years ago

Also, to clarify -- I ran into this in an entirely fresh profile, where I hadn't toggled any settings at all. (I didn't set Tracking Protection to "Never" - I'm guessing maybe the shield study toggled a pref to do that, and then is adding its own layer of enabledness on top of that? But that creates this contradictory UI issue, unfortunately.)

dholbert commented 6 years ago

Screenshot of shield study icon telling me that Tracking Protection is "Enabled": screenshot of shield icon

Screenshot of Firefox preferences contradicting that: screenshot of preferences page

biancadanforth commented 6 years ago

Thank you @dholbert for bringing this up and for the clear description of the issue.

This is expected behavior for this Shield study, as the built-in Tracking Protection feature in Firefox does not currently have a way to access the number of blocked requests per page. Since we wanted to make that information available to users for this study, we had to disable built-in Tracking Protection and re-implement it using the WebRequest.jsm module in Firefox.

Rest assured trackers are being blocked while you are enrolled in the study.

Here is some more information about it in the comments of the study’s main module, Feature.jsm, and here is where the study blocks the tracker requests.

dholbert commented 6 years ago

Thanks @biancadanforth! Yeah, I was pretty sure that trackers were being blocked despite the contradictory UI (thanks for the links to confirm) -- I just wish we could gray out the basically-incorrect UI in the Firefox preferences page, or add an "overridden via Shield Study" annotation/tooltip, or do something to avoid presenting incorrect/confusing UI.

As it stands right now, I suspect some nontrivial fraction of study-recipients will be curious about Tracking Protection (because we're prompting them about it), and they'll go looking for more information/knobs in our preferences, only to land on the confusing & basically-incorrect "Use Tracking Protection: Never" chunk that I screenshotted above. And then they'll end up confused & perhaps just shrug it off & lose a little confidence in the accuracy of Firefox's blocking tools.

(I suspect there's nothing we can do at this point, and maybe Shield Studies don't even have the capability to mess with our preferences UI... but it'd be nice if we could build something like this into future Shield studies that would otherwrise have similar contradictory messaging, if possible.)

biancadanforth commented 6 years ago

Thanks @dholbert for the thoughts. You're right that it would be less confusing if we could update the Preferences page in some way.

Part of the challenge is that we don't want users' awareness that they are in a study to influence what would otherwise be their normal behavior (this is known, I just discovered, as the "Hawthorne Effect" or "observer effect").

This was definitely a weakness of the study, and it's worth considering how we can improve for similar studies that look at/modify existing features in Firefox in the future.