privacytests / privacytests.org

Source code for privacytests.org. Includes browser testing code and site rendering.
https://privacytests.org
MIT License
798 stars 23 forks source link

some state partitioning results are ambiguous #91

Open Thorin-Oakenpants opened 2 years ago

Thorin-Oakenpants commented 2 years ago

LW, for example, uses session cookies (cookie lifetime pref), and SWers will fail, so it's not enough to check navigator keys

arthuredelstein commented 2 years ago

Thanks! Could you provide more details? I don't fully understand. :) Do you think it should be a fail for LibreWolf, or just a "not supported" or something?

Thorin-Oakenpants commented 2 years ago

At least in FF, cookie permission controls access to most/all "site data": localStorage, sessionStorage, IDB, sharedWorkers, serviceWorkers (and thus SW cache). For example if you block the cookies, all the above will fail

Firefox has a UI setting > about:preferences#privacy > Cookies and site data > Delete .. when closed

In at least FF91+ (in earlier versions sharedWorkers also had this issue, but I'm not going to track down the change), this breaks serviceWorkers. i.e SWers are in navigator, but they will throw a SecurityError. In other words, you can have SWers enabled, but you still need to trap when they fail, otherwise you're not really testing that they are partitioned

clicking on LW's SW test you can see you return Error: The operation is insecure. So in this case I would trap errors and return = no such feature

Thorin-Oakenpants commented 2 years ago

FYI:

Thorin-Oakenpants commented 2 years ago

renamed the issue, and I will expand on this. What does a green check mark mean? Currently the legend says "passed privacy check". For many tests, the result is not a successful "partitioning", but rather 3rd party contexts being denied. So while technically this is true (privacy was preserved), it also fails to show a difference between those that actually partition, and those who limit.

The State Partitioning blurb says (emphasis mine)

A common vulnerability of web browsers is that they allow tracking companies to 'tag' your browser with some data ('state') that identifies you. When third-party trackers are embedded in websites, they can see this identifying data as you browse to different websites. Fortunately, it is possible for this category of leaks to be fixed by partitioning all data stored in the browser such that no data can be shared between websites.

Interpreting the results will depend on how each test is done, but I believe it is important to differentiate here between those that actually partition and para-quoting "fix this category" vs those that do not