search

arkenfox / user.js

Firefox privacy, security and anti-tracking: a comprehensive user.js template for configuration and hardening
MIT License
10.04k stars 514 forks source link

discussion: persistent storage strategies [cookies, indexeddb + local dom] #188

Closed Thorin-Oakenpants closed 7 years ago

Thorin-Oakenpants commented 7 years ago

Currently we have network.cookie.cookieBehavior set as 2 with a mention that You can set exceptions under site permissions or use an extension

0=allow all 1=allow same host 2=disallow all 3=allow 3rd party if it already set a cookie

We could change this to 1 as it would fix some underlying issues (but maybe raise some).

Why can't ALL cookies be accessible in ALL modes (PB, FPI, TP+SB), as well as local dom storage

So, as you can see .. downsides, but upsides

Class, discuss!

Thorin-Oakenpants commented 7 years ago

With FPI and PB mode - cookie addons are useless for cleaning. They are OK for blocking (see uMatrix) - so enabling cookies is FINE - my concern is the local storage

Atavic commented 7 years ago

3=allow 3rd party if it already set a cookie

Does it still work? Seems abandonware to me.

Thorin-Oakenpants commented 7 years ago

^^ no idea, probably means nothing in the context of FPI as per domain (origin) will cause multiples of it. We're not even considering it anyway, so who cares :)

Edit: unless you meant that the value of 3 does not do what we say it does?

Thorin-Oakenpants commented 7 years ago

FYI: https://bugzilla.mozilla.org/show_bug.cgi?id=1362834 - protected SB google cookie https://bugzilla.mozilla.org/show_bug.cgi?id=1381197 - FPI & cookies

Thorin-Oakenpants commented 7 years ago

PS: I suggest if you are using FPI to delete all your cookies and let them get re-created so they have the correct attributes

Just-me-ghacks commented 7 years ago

This is what I use: user_pref("network.cookie.cookieBehavior", 1); user_pref("dom.storage.enabled", false);

I don't block ALL cookies with uMatrix. Should I?

Occasionally I have to enable dom storage but after that I delete my entire Firefox folder.

I haven't seen any "SB google cookie". Ever. I have all the SB stuff disabled. Where should I check for this cookie?

Thorin-Oakenpants commented 7 years ago

I haven't seen any "SB google cookie". Ever. I have all the SB stuff disabled. Where should I check for this cookie?

AFAIK, if you have it, it should be visible under about:preferences#privacy > View Cookies - see https://bugzilla.mozilla.org/show_bug.cgi?id=1362834

I don't block ALL cookies with uMatrix. Should I?

Why not? Use uM exceptions. Currently I have to allow an exception in both FF's site permissions AND uM, so setting cookies as 1st party removes a step - that is NOT the issue. The question is not really about cookies but about dom storage (which is in effect controlled by FF's internal cookie setting).

When I ditched Cookie Controller, I removed all my cookies, removed all cookie exceptions, FPI is on, and recreated site exceptions, created exceptions in uM, and recreated the cookies by visiting the sites (this to make sure all cookies are new with the right attributes etc) - seriously .. I now have a grand total of 7 cookies and 1 session cookie if it exists) = 8 sites total. Why bother allowing cookies by default. A whitelist saves so much BS. A block all also means much better control over them due to FPI & PB mode issues re extensions.

The equation is (remembering that the cookie setting also controls dom storage)

Either (network.cookie.cookieBehavior, dom.storage.enabled)

  1. cookie :x: dom storage :white_check_mark: (current)
    • means simply allow site exceptions (eg allow, allow for session) for sites that need dom (I have to admit I see very very little of this requirement with my web browsing) or a cookie for logins
    • You could still block the cookie itself (eg uMatrix) but the firefox internal pref means dom is granted (pretty sure uM blocking the cookie does not affect local dom = needs testing to confirm)
    • You still need to clear dom (for the few sites it gets set)
  2. cookie :white_check_mark: dom storage :white_check_mark:
    • by allowing cookies this allows dom storage, even if cookies are blocked in uM (needs test). This makes websites work = less breakage, but also allows WebExtensions that use local storage to also work (uBo may be one of these) :+1:
    • you still need to clear dom (but now its all sites, not a few) :-1:
    • note that FPI means cookies & dom storage are isolated which is good, but they are also re-used by their own first parties and can track you when reloading the site. This has always been the case
  3. cookie :x: dom storage :x:
    • idk
  4. cookie :white_check_mark: dom storage :x:
    • idk

Under site prefs is an option to Maintain Online Storage. IF this directly relates to dom storage, then the solution is number 4 - use site exceptions for cookies in uM with (default deny-all in uM) and FF site exceptions for dom storage to offer the perfect granular control - EDIT: But this would break web extensions and I can't see how you can grant an exception for those, and that was the whole point of this exercise

earthlng commented 7 years ago

I'm confused about your use of "dom storage" - What is that? I think you're mixing them up.

What kind of offline storage exists and where is it stored should be the first thing we clarify.

Afaik, there's

Is there anything else? Which ones are you worried about? DOM storage is something completely different, basically variables, objects, etc that a sites JS defines and are temporarily stored in DOM.

Thorin-Oakenpants commented 7 years ago

^^^^^ THIS one

/* 2705: disable DOM (Document Object Model) Storage
 * [WARNING] This will break a LOT of sites' functionality.
 * You are better off using an extension for more granular control ***/
   // user_pref("dom.storage.enabled", false);

THIS is called local dom storage by almost everything I know <<< ELI5 WHERE THIS LIVES

Thorin-Oakenpants commented 7 years ago

@fmarier : can you confirm that all identifying information in any SB google cookie is stripped out. I do not have one to look at. Is it a requirement (ToS) that a SB cookie even be used, because I update just fine and I block it

2glops commented 7 years ago

Found that test page from MDN : http://dev-test.nemikor.com/web-storage/support-test/

Even with dom.storage.enabled=true, I notice : localStorage: not supported sessionStorage: limited to 0 k characters globalStorage: not supported

Thorin-Oakenpants commented 7 years ago

^^ @2glops - did you allow a site exception for cookies for that test site? I disabled uM and uBo and NS temporarily for that site, added a cookie exception (site prefs) and clicked run the tests and nothing seems to happen.

2glops commented 7 years ago

With site exception cookie (about:prefs), that returns : localStorage: limited to 5101 k characters sessionStorage: limited to 5101 k characters globalStorage: not supported

The tests seems to have completed (5 min run) but no dialog to confirm)

Thorin-Oakenpants commented 7 years ago

Right, which is exactly what I have been saying, that the FF pref for cookies also controls the local storage doesn't matter if it is global or a site exception.

I got the same result with eCleaner - even though it doesn't need or set a cookie, it requires cookies enabled (0 or 1), as then allows local dom storage, which in turn meant the Web Ext could access it's saved settings

We need a test where a cookie is set and it uses local storage. So we can allow the site a cookie in FF, but block the cookie in uM, and see if local dom is still accessible

Thorin-Oakenpants commented 7 years ago

This is interesting: https://addons.mozilla.org/en-US/firefox/addon/shim-storage/ (legacy addon - very old, 4 years abandoned but still works AFAIK from a comment on redditttt)

If your disabling localStorage or sessionStorage many sites stops working. Not cause they actually need the database, but cause they expect it to be there.

This extension allows you to define a list of domains where a shim should be defined so the page will actually work. No data will be stored anywhere outside the page.

NOTE: This extension has no effect if you don't set dom.storage.enabled to false.

So in effect:

  1. Allow cookies in FF (which in effect NOW stops blocking dom storage) but block all cookies in uMatrix by default. uM giving you granular control
  2. Deny dom.storage.enabled and use the shim storage extension for granular control

Wonder how hard it would be or if even possible in a WebExt? However, this still doesn't allow exceptions for Web Extensions local storage AFAIK, which is a problem. But that aside it does give some granular control for some problem sites - the other drawback is sites that actually need it will fail.

Just about ready to give up. I hope the new about:permissions gives a much more precise and thorough configuration (eg indexedb, dom storage, offline storage, cookies = and none are dependent on others, and extensions are exempt)

Atavic commented 7 years ago

Re: https://github.com/ghacksuserjs/ghacks-user.js/issues/188#issuecomment-320470847

3=allow 3rd party had it's own policy (P3P) that was taken down by google, inc. et al.

earthlng commented 7 years ago

Hey Pants, don't you get a bunch of error messages in the console after startup in 55? FF also uses indexedDB internally and disabling it might have unforeseen consequences. I've now enabled it again and will use PB windows to prevent sites from using it ie. if a site needs JS I'll make sure to open it in PB.

As for "Web Storage" (aka DOM Storage) uMatrix has an option to delete it for blocked hostnames. testpage: https://mdn.github.io/dom-examples/web-storage/ Information is stored in the webappsstore.sqlite file for websites and in the chromeappsstore.sqlite for [about:*] (source)

I'm using 1 as the cookie value (no 3rd party cookies), indexedDB enabled, dom.storage.enabled=true and clear cookies on shutdown which also deletes sites "localstorage".

Thorin-Oakenpants commented 7 years ago

Looking in my "D:\Portable\FirefoxPortable\Data\profile\storage\permanent\" (which I assume is FF internals), I see date modified is all like 2 years old - heh

earthlng commented 7 years ago

You don't clean up your profile folder very much then, huh? xD

Why would mozilla provide an indexedDB pref that breaks INTERNAL code.

IDK but they do.

How can I see the console error messages when I start? EIL5 please.

I have to ELI5 a lot recently ;) start FF then Ctrl+Shift+J

Thorin-Oakenpants commented 7 years ago

You don't clean up your profile folder very much then, huh

I do. There's nothing in there. ...\profile\storage\default and ...\profile\storage\temporary are empty ...\profile\storage\permanent has three sub-dirs: chrome, indexeddb+++fx-devtools and moz-safe-about+home and these are internal, so I don't care about cleaning them

I guess its been about 2 years since I toggled indexedDB on. My profile is well over 4 years old. I keep it clean as I can (removed dead prefs including extension ones).

Maybe with 57 I will start a brand new clean profile.

Thorin-Oakenpants commented 7 years ago

FYI - nilla 55 profile with indexedDB disabled

noindexeddb

AFAIK, this isn't affecting ANYTHING I care about

earthlng commented 7 years ago

The 3rd one from abouthome I don't care about either. The other 4, afaik, are 2 each for something to do with devtools and addons. As I said earlier "it might have unforeseen consequences". Idk if or how this affects devtools and/or addons.

Thorin-Oakenpants commented 7 years ago

From Atavic

3=allow 3rd party if it already set a cookie Does it still work? Seems abandonware to me

 0=allow all 1=allow same host 2=disallow all 3=allow 3rd party if it already set a cookie
 [SETTING] Options>Privacy>History>Custom Settings>Accept cookies from sites

@Atavic :

This allows 4 outcomes

So that maps nice and correct to what we said in the user.js

Thorin-Oakenpants commented 7 years ago

For extensions that "require" cookies (they don't actually need them) such as Stylus, you can set an extension exception by going to about:debugging, selecting the manifest URL and from there adding your exception from the urlbar info icon etc.

It's looking more and more like cookies will need to be unblocked at the core FF level (1st party only), and blocked by eg uMatrix. I am OK with this.... BUT uMatrix will then only block the cookie itself and now every single site will have access to local dom storage (right?). This creates a problem - eg auto-cleaning local dom storage eg via Cookie-AutoDelete - i.e, no cookie, no detection of dom to clear (note that the bugs that prevented cleaning dom have and are being resolved fixed).

@earthlng - can you please tell me, yes or no, the answer to this: Q: If I allow cookies in FF Options, but block them using uM, can a website still set local dom storage

Atavic commented 7 years ago

Gorhill's Efficient blacklisting:

Blacklisted cookies are not prevented by uMatrix from entering your browser. However they are prevented from leaving your browser , which is what really matters.

(from wiki)

Thorin-Oakenpants commented 7 years ago

Ahh, well that changes things - clearly local dom storage is opened up as well. So it then becomes an issue when not in default-deny to take care all all the fallout of cookies + dom and cleaning them out What a mess. I have loved the last 5 years without having to clean hardly anything.

Maybe the extension exclusion will be enough. I guess I'll find out as I go along. Meanwhile, indexeddb I might put system permission locks on ...\storage\default and ...\storage\temporary - although 4 or 5 days into allowing indexeddb, there is still nothing in there

earthlng commented 7 years ago

@earthlng - can you please tell me, yes or no, the answer to this: Q: If I allow cookies in FF Options, but block them using uM, can a website still set local dom storage

Yes but you can enable "Delete local storage content set by blocked hostnames".

Mozilla has setup a nice demo page where you can test it yourself. If you don't allow the cookie for that page in uMatrix you can see in uM's logger that localstorage is blocked: -- cookie https://mdn.github.io/{localStorage}

It's a bit weird though because you can change the values on that page and a normal refresh of the page will still use those values. However when you shift+refresh the page it resets to the red background and the default values. For me that's good enough. But that's with the old non-WE uMatrix! They are still working on allowing WE's to clear localstorage for certain hosts only.

Thorin-Oakenpants commented 7 years ago

Can't be bothered reading up and searching, not sure if anyone said indexeddb was cleared on close with anything .. (clue, it's not) - here is a demo page: https://static.raymondcamden.com/demos/2014/feb/7/index.html#/home (not that I tried it)

Here is a 1340511 for extend browsingData api for single site - that is the ability to remove site data per domain (cookies, localStorage, indexedDB, serviceWorkers, cacheStorage, applicationCache)

Here is 1333050 which is one step of the above, clearing of IndexedDB in browsingData API

crssi commented 7 years ago

^^The demo page is using default storage in profile/storage/default folder location. This is persistent and it does not get cleared with any to me known preference or addon and I have all kind of clearing mechanisms in place. The only avoiding way as I see right now is to delete default and temporary folders and make files with the same names (file name extensionless) to block it. But in that case adding a node on this demo page doesn't even work as intended.

Is there anyway ATM to clean those folders with a preferences or addon?

Thorin-Oakenpants commented 7 years ago

No. That's why I linked to 1333050 ... indexedDB has never really had any access to my knowledge and is a real problem

Edit: I've had indexeddb on for about 10 days I think, yet to see ANYTHING land on my hdd. That said, you can trick sites by allowing indexedDB and denying permissions to the folders, and this then makes them work in some cases, IDK exactly - read https://www.wilderssecurity.com/threads/firefox-and-indexeddb.395113/ first comment

crssi commented 7 years ago

I guessed so. Making those folders read only can be tricky, since FF update could "repair" ACLs on those and better solution would be to remove those and create files with the same name. I have done so ages back and I can't remember if any site broke because of it... with an exception of here mentioned demo site.

crssi commented 7 years ago

The demo site doesn't work in PB. Also I loose control over cookies in PB.

Forsaked commented 7 years ago

@crssi Any Outlook Web App would break without IndexedDB enabled.

crssi commented 7 years ago

@Forsaked see https://github.com/ghacksuserjs/ghacks-user.js/issues/180#issuecomment-317660941 It goes even further back... see https://github.com/ghacksuserjs/ghacks-user.js/issues/17 ;) And I have IndexDB enabled long time ago, never actually disabled it (only for testing). But it can be enabled and with preventing FF to write to storage OWA still works like numerous other sites.

crssi commented 7 years ago

@earthlng no it does not... you made me retest with a clean profile. :) it opens, but doesn't work as intended... try click on "Add Node" green button on the right... nothing happens in PB and the same result in normal browsing with IndexDB disabled or IndexDB enabled with blocked storage (default) on file level.

crssi commented 7 years ago

:) In my configuration with normal browsing FF doesn't remember anything with test site you mentioned without needing to restart FF. Which is exactly what I want.

earthlng commented 7 years ago

Bug 781982 - IndexedDB does not function in private browsing mode

bostonglobe.com seems to be using this to detect private browsing: https://pastebin.com/bphxwrtT

Thorin-Oakenpants commented 7 years ago

Now that's funny. One of PB mode's purposes is to not leave persistent data on close, and because there is no way for FF to clean indexedDB, they just left disabled it. @fmarier - that's not a solution. Breaking sites in PB mode like that. Don't you get telemetry analysis bringing these things up? I would have expected thousands of complaints that PB mode breaks sites. Is there any telemetry on % of sites that use indexedDB?

fmarier commented 7 years ago

Don't you get telemetry analysis bringing these things up?

We have a policy of not collecting any telemetry in Private Browsing mode. That might change once that RAPPOR system is in place, but right now we know very little about what goes on in Private Browsing.

Thorin-Oakenpants commented 7 years ago

Doh! Of course. I like that policy :+1: Don't change it EXCEPT if you do, limit it to Nightly

Thorin-Oakenpants commented 7 years ago

Looks like 1388428 - Extend browsingData to support removing localStorage by host isn't far away, which will allow C-AD to gain "parity" with legacy versions. Still a problem for cookies+localstorage set with FPI though which is the deal breaker for me, i.e all cookies need to be denied by default.

Thorin-Oakenpants commented 7 years ago

Hmmm .. I block all cookies by default (I actually only allow around 8 domains as 1st party, that's it). This also blocks sites from local dom storage. I wonder if this is also blocking indexedDB .. because after almost a month of having indexedDB enabled, I have yet to get ANYTHING put in there from web content - and I visited sites like YT, NYT, whatever .. and I get absolutely no data in my /storage/default+temporary folders .. not a bean .. zip, zilch, zero, nuttin', nothing, nada.

Might do some more testing in a nilla

Thorin-Oakenpants commented 7 years ago

Offline Storage: I knew that was dodgy, I think Tom told me - anyway : https://bugzilla.mozilla.org/show_bug.cgi?id=1077986

earthlng commented 7 years ago

Your problem with cookies + localstorage only arises because you want to permanently keep those 8 cookies. If you could live without that, the easiest solution would be to allow cookies in FF, optionally block them with uMatrix and clear cookies on shutdown (which also clears localstorage). That's at least what I do. It also has the advantage that I don't need to tamper with FF's site exceptions/permissions which IMO suck anyway because there's no easy way to see all the sites for which a permission was set/changed (afaik). And it would solve the problems that a bunch of people reported here because cookies were/are blocked by our template. Just my 2 cents

Thorin-Oakenpants commented 7 years ago

The removal of about:permissions was a blow. We used to be able to see that info

There will be different methods that work for each of us. Me, I'm a lone wolf, zero access to my lair/machine - I like my auto-login for 5 sites (and 3 cookies for site prefs) - these 8 sites I am ok with keeping. There is no 3rd party leakage.

The problem is most of the persistent storage APIs are broken. Clearing history has issues, IDB is a nightmare, and so on. WHEN these are fixed, there should be multiple solutions that work for everyone

Thorin-Oakenpants commented 7 years ago

PS: I can look up my site permission cookie exceptions by looking in Options>Privacy>Show Cookies

EDIT: Actually, you can just click Exceptions and it show all the site exceptions as Allow or Allow for Session.

I don't think I have any other types of exceptions, but it would be nice for the new about:permissions for each type of exception to be able to look them up (eg all sites with override full screen block etc)

curiosity-seeker commented 7 years ago

I've had indexeddb on for about 10 days I think, yet to see ANYTHING land on my hdd. That said, you can trick sites by allowing indexedDB and denying permissions to the folders, and this then makes them work in some cases, IDK exactly - read https://www.wilderssecurity.com/threads/firefox-and-indexeddb.395113/ first comment

One example is https://www.gmx.net. That site ceates a new entry in ...storage/default.

Denying write access to the whole default folder does not work unfortunately as several webextensions save data therein (moz-extension+++ .... subfolders). There are also other subfolders (about+newtab ...) possibly related to containers ?

Anyway, my current strategy is the following:

  1. Delete all subfolders in default pointing to specific domains.
  2. Go up to the storage folder and remove write permission for default: chmod -R ug-w default
  3. Add write permission for all existing subfolders in default beginning with "moz" and "about": 
    find ./default/ -name "moz*" -exec chmod -R ug+w {} +
find ./default/ -name "about*" -exec chmod -R ug+w {} +

    These are Linux commands. I don't know if anything similar is possible for NTFS permissions (but it can certainly be done manually). This works well for me. One disadvantage is that newly installed add-ons are not allowed to save anything in storage. In this case write permission must be re-added for default and later on above steps must be repeated.

Let's hope that the Firefox devs come up with a better solution. Unfortunately deleting website data in about:preferences#privacy also deletes all cookies - including the ones you specifically allowed for, e.g., searx.me or startpage in order to save your search settings.

curiosity-seeker commented 7 years ago

Okay - forget what I wrote in the previous post: The issue is fixed in FF58 Nightly which I got today. Obviously related to the fixes mentioned by Pants here.

Thorin-Oakenpants commented 7 years ago

^^ https://bugzilla.mozilla.org/show_bug.cgi?id=1047098 - pushed to 56. Told ya one of them was a block. This will also make it to ESR I believe

EDIT: and the other one was pushed back to 57

atomGit commented 7 years ago

IndexDB - /storage/default

what i'm seeing is that, if you set uM to the global scope and block "other" in the "all" row and then visit something like YouTube, no local storage object gets created (even if allowed in FF permissions UI (default))

to me this is a bit odd because there are no "other" entries in uM for youtube.com - i don't know why uM doesn't handle IndexDB storage better - even if you block cookies in the global scope, the local storage still gets created, as well as cookies - i understand uM doesn't block cookies by default, but rather just prevents them from leaving the browser, but still, if you purposely set it to block then it damn well should

as for automating the removal of local storage objects, SDC is dead and the only other viable alt i'm seeing is Cookie AutoDelete (webext), but it doesn't yet handle local storage

Cookies Exterminator may work, but it's legacy